Image recognition method and device

Abstract

The invention discloses an image recognition method and device, and the method comprises the steps: obtaining a to-be-processed image needing to be processed and the shooting information of the to-be-processed image; performing feature extraction on the to-be-processed image to obtain image features; coding the shooting information to obtain shooting features; fusing the image features and the shooting features to obtain multi-modal features; and determining an image recognition result of the to-be-processed image based on the multi-modal features. According to the method, the image features and the shooting features of the to-be-processed image are comprehensively utilized to obtain the multi-modal features including the image features and the shooting features, then the multi-modal features are utilized to perform image recognition, and the image recognition result highly related to the real shooting features can be generated according to the multi-modal features, so that the accuracy of image recognition can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image recognition method and device. Background Art

[0002] With the continuous development of information technology and artificial intelligence technology, many achievements have been made in the field of multimodal learning, and it has been widely used in actual production and life, bringing convenience to mankind, such as intelligent detection of medical images, smart city systems, image recognition and recommendation of e-commerce software, etc. Among them, the implementation of image caption generation related technologies plays a key role, but the image caption generation in the existing technology does not take into account the parameters when the image is taken, resulting in low accuracy of image recognition.

[0003] That is, the accuracy of the image recognition method in the prior art is low. Summary of the invention

[0004] The embodiments of the present application provide an image recognition method and device, which can improve the accuracy of the image recognition method.

[0005] In a first aspect, the image recognition method provided by the present application comprises:

[0006] Acquire an image to be processed and shooting information of the image to be processed;

[0007] Extracting features of the image to be processed to obtain image features;

[0008] Encoding the shooting information to obtain shooting features;

[0009] fusing the image feature and the shooting feature to obtain a multimodal feature;

[0010] An image recognition result of the image to be processed is determined based on the multimodal features.

[0011] In a second aspect, the image recognition device provided by the present application includes:

[0012] An acquisition module, used for acquiring an image to be processed and shooting information of the image to be processed;

[0013] A feature extraction module is used to extract features from the image to be processed to obtain image features;

[0014] An encoding module, used for encoding the shooting information to obtain shooting features;

[0015] A fusion module, used for fusing the image feature and the shooting feature to obtain a multimodal feature;

[0016] A recognition module is used to determine the image recognition result of the image to be processed based on the multimodal features.

[0017] In a third aspect, the electronic device provided in the present application includes a memory and a processor, the memory stores a computer program, and the processor is used to run the computer program in the memory to implement the steps in the image recognition method provided in the present application.

[0018] In a fourth aspect, the computer-readable storage medium provided in the present application stores a plurality of instructions, which are suitable for loading by a processor to implement the steps in the image recognition method provided in the present application.

[0019] In a fifth aspect, the computer program product provided in the present application includes a computer program or instructions, which, when executed by a processor, implement the steps in the image recognition method provided in the present application.

[0020] In this application, compared with the related art, the image to be processed and the shooting information of the image to be processed are obtained; the image to be processed is subjected to feature extraction to obtain image features; the shooting information is encoded to obtain shooting features; the image features and the shooting features are fused to obtain multimodal features; and the image recognition result of the image to be processed is determined based on the multimodal features. This application comprehensively utilizes the image features and shooting features of the image to be processed to obtain multimodal features including image features and shooting features, and then uses the multimodal features for image recognition, which can generate image recognition results that are highly correlated with the actual shooting features based on the multimodal features, thereby improving the accuracy of image recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 is a schematic diagram of a scene of an image recognition system provided by an embodiment of the present application;

[0023] Figure 2 It is a flowchart of an embodiment of the image recognition method provided by the embodiment of the present application;

[0024] Figure 3 is a flowchart of another embodiment of the image recognition method provided by the embodiment of the present application;

[0025] Figure 4 is a schematic diagram of the structure of an image recognition device provided in an embodiment of the present application;

[0026] Figure 5 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] It should be noted that the principles of the present application are illustrated by implementing them in an appropriate computing environment. The following description is based on the illustrated specific embodiments of the present application and should not be considered as limiting other specific embodiments of the present application that are not described in detail herein.

[0028] In the following description of the present application, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0029] In the following description of the present application, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0031] In order to improve the effect of image recognition, the embodiments of the present application provide an image recognition method, an image recognition device, an electronic device, a computer-readable storage medium, and a computer program product. The image recognition method can be executed by the image recognition device, or by an electronic device integrated with the image recognition device.

[0032] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0033] Please refer to Figure 1 , the present application also provides an image recognition system, such as Figure 1 As shown, the image recognition system includes an electronic device 100, and the electronic device 100 is integrated with the image recognition device provided by the present application.

[0034] Among them, the electronic device 100 can be any device equipped with a processor and has processing capabilities, such as mobile electronic devices with processors such as smart phones, tablet computers, PDAs, laptops, smart speakers, or fixed electronic devices with processors such as desktop computers, televisions, servers, industrial equipment, etc.

[0035] In addition, if Figure 1 As shown, the image recognition system may further include a memory 200 for storing original data, intermediate data, and result data.

[0036] In the embodiment of the present application, the memory 200 can be a cloud storage. Cloud storage is a new concept extended and developed from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as the storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of different types of storage devices (storage devices are also called storage nodes) in the network through application software or application interfaces to work together and provide external data storage and business access functions.

[0037] At present, the storage method of the storage system is: create a logical volume, and when creating a logical volume, allocate physical storage space for each logical volume. The physical storage space may be composed of disks of a storage device or several storage devices. The client stores data on a logical volume, that is, stores the data on the file system. The file system divides the data into many parts, each of which is an object. The object contains not only data but also additional information such as data identification (ID, ID entity). The file system writes each object into the physical storage space of the logical volume, and the file system records the storage location information of each object, so that when the client requests to access the data, the file system can allow the client to access the data according to the storage location information of each object.

[0038] The process of the storage system allocating physical storage space to a logical volume is as follows: based on the estimated capacity of the objects stored in the logical volume (this estimate often has a large margin relative to the actual capacity of the objects to be stored) and the grouping of independent redundant disk arrays (RAID, Redundant Array of Independent Disks), the physical storage space is pre-divided into stripes. A logical volume can be understood as a stripe, thereby allocating physical storage space to the logical volume.

[0039] It should be noted that Figure 1The scene diagram of the image recognition system shown is only an example. The image recognition system and scene described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of the image recognition system and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0040] It should be noted that the serial numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0041] Please refer to Figure 2 , Figure 2 FIG. 1 is a flow chart of an embodiment of the image recognition method provided in the present application. Figure 2 As shown, the process of the image recognition method provided by this application is as follows:

[0042] 201. Obtain an image to be processed and shooting information of the image to be processed.

[0043] In the embodiment of the present application, the image to be processed may be an image uploaded by the user or an image acquired by a camera, and the shooting information of the image to be processed may include shooting time information, shooting location information, shooting scene information, etc., which can be set according to the specific situation. For example, the shooting time information is 13:14 on January 1, 22; the shooting location information is CC Street, BB District, AA City.

[0044] In a specific embodiment, a user uploads an image to be processed and shooting information of the image to be processed, and obtains the image to be processed and shooting information of the image to be processed.

[0045] In another specific embodiment, obtaining the to-be-processed image and the shooting information of the to-be-processed image includes:

[0046] (1) Obtain the image to be processed.

[0047] In the embodiment of the present application, an image to be processed uploaded by a user is obtained.

[0048] (2) Obtain metadata of the image to be processed.

[0049] The metadata of an image is some tags embedded in the image file. The metadata of the image to be processed can be metadata in the Exchangeable Image File Format (EXIF), metadata in the International Press Telecommunications Council (IPTC) format, or metadata in the Extensible Metadata Platform (XMP) format. EXIF ​​metadata is specially set for photos taken by digital cameras and can record the attribute information and shooting data of digital photos. EXIF ​​is usually automatically added by digital cameras when taking photos, such as camera model, lens, exposure, image size and other information. IPTC metadata is a standard format that can add the following metadata to photo information, such as author, copyright, subtitles, detailed descriptions, etc. XMP metadata is actually a standard for metadata storage and management, which can store EXIF, IPTC or other data in image files in the unified format of XMP. The way to embed metadata varies depending on the image format, and image files of different formats have different embedding methods.

[0050] Specifically, before obtaining the metadata of the image to be processed, the image obtained by taking the photo and the metadata corresponding to the image are stored in the album when taking the photo. The metadata corresponding to the image includes the shooting time and the shooting location. The shooting location can be obtained through the Global Positioning System (GPS) or the Beidou system.

[0051] (3) Extracting the shooting information of the image to be processed from the metadata of the image to be processed.

[0052] 202. Perform feature extraction on the image to be processed to obtain image features.

[0053] In an embodiment of the present application, the image to be processed is input into a preset feature extraction model to obtain image features. The preset feature extraction model can be a Faster RCNN model or a CLIP (Contrastive Language-Image Pre-training) model, etc., which can be set according to the specific situation. The CLIP model is a pre-trained neural network model for matching images and texts released by OpenAI in early 2021. It can be said to be a classic in the field of multimodal research in recent years. The model directly uses a large amount of Internet data for pre-training, and has achieved the current best performance in many task performances.

[0054] In a specific embodiment, the CLIP model is used to extract features from the image to be processed to obtain image features in a two-dimensional array format. For example, the image features are a two-dimensional array of [50,512].

[0055] It should be noted that artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0056] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes machine learning (ML) technology, among which deep learning (DL) is a new research direction in machine learning. It is introduced into machine learning to make it closer to the original goal, namely artificial intelligence. At present, deep learning is mainly used in the fields of machine vision and natural speech processing.

[0057] Deep learning is the process of learning the inherent laws and representation levels of sample data. The information obtained in the learning process is very helpful for interpreting data such as text, images, and sound. Using deep learning technology and corresponding sample sets, network models that can achieve different functions can be trained.

[0058] 203. Encode the shooting information to obtain shooting features.

[0059] In an embodiment of the present application, the shooting information of the image to be processed may include shooting time information and shooting location information. For example, the shooting time information is 13:14 on January 1, 22; the shooting location information is CC Street, BB District, AA City. The shooting features may include target time coding features and target location coding features. The target time coding features include a preset number of dimensions. Of course, the shooting information of the image to be processed may only include shooting time information, and the shooting features only include target time coding features; the shooting information of the image to be processed may also only include shooting location information, and the shooting features only include target location coding features, which can be set according to the specific situation.

[0060] In an embodiment of the present application, in order to obtain accurately extracted target time coding features, the shooting information is encoded to obtain shooting features, including: obtaining a preset number of time fields in the shooting time information; inputting a preset number of time fields in the shooting time information into a preset convolutional neural network for encoding to obtain target time coding features, and the target time coding features include a preset number of dimensions.

[0061] Among them, the preset convolutional neural network can be a neural network such as RNN, CNN, transformer, etc., which is not limited here. The preset number t can be set according to the specific situation, and the time field of the preset number t can be set according to the specific situation. For example, the preset number t is 5, and the fields of the preset number t are year, month, day, hour and minute respectively. The shooting time information is 13:14:15 on January 1, 22. The preset number of time fields in the shooting time information is obtained, and the 5 time fields are 22, 1, 1, 13, and 14 respectively. The 5 time fields are input into the preset convolutional neural network for encoding to obtain the target time coding feature. The target time coding feature includes 5 dimensions, and the characteristic values ​​of the 5 dimensions of the target time coding feature are [22, 01, 01, 13, 14] respectively.

[0062] In an embodiment of the present application, in order to obtain accurate extraction of target position coding features, the shooting information is encoded to obtain shooting features, including: encoding the shooting position information according to a preset character set to obtain initial position coding features; inputting the initial position coding features into a preset convolutional neural network for encoding to obtain target position coding features.

[0063] Among them, the preset character set can be a character set of Unicode, which is also called the Universal Code or Single Code. It is developed by the Unicode Consortium and is an industry standard in the field of computer science, including character sets, encoding schemes, etc. Of course, the preset character set can also be an encoding using the American Standard Code for Information Interchange (ASCII) character set or Arabic numerals and Chinese characters. The American Standard Code for Information Interchange is a computer encoding system based on the Latin alphabet, mainly used to display modern English and other Western European languages. ASCII was first published in the form of a standard in 1967, and the last update was in 1986. So far, a total of 128 characters have been defined.

[0064] 204. The image features and the shooting features are integrated to obtain multimodal features.

[0065] Specifically, the image features, target time coding features and / or target position coding features are spliced ​​in a preset arrangement order to obtain multimodal features. The preset arrangement order can be set according to specific conditions. For example, the preset arrangement order is to arrange the image features, target time coding features and target position coding features in sequence. For example, the image features are a two-dimensional array of [x, m], the dimension of the target time coding features is t, and the dimension of the target position coding features is l. Then, the image features, the target time coding features and the target position coding features are spliced ​​in a preset arrangement order to obtain multimodal features, and the multimodal features are a two-dimensional array of [x+t+l, m]. Because the feature arrangement order and format of the multimodal features are restricted, the gradient of the model can decrease faster during training, thereby improving learning efficiency.

[0066] 205. Determine an image recognition result of the image to be processed based on the multimodal features.

[0067] The image recognition result may be the description information of the image to be processed. For example, the image recognition result of an image to be processed may be “a small bird resting on a green tree”.

[0068] In a specific embodiment, the multimodal features are input into a pre-trained preset multimodal image description model to obtain an image recognition result of the image to be processed.

[0069] In another specific embodiment, in order to improve the prediction accuracy, the image recognition result of the image to be processed is determined based on the multimodal features, including: determining the corresponding target multimodal image description model based on the feature combination in the multimodal features, wherein different feature combinations correspond to different multimodal image description models; and inputting the multimodal features into the target multimodal image description model to obtain the image recognition result of the image to be processed. Since different feature combinations use different multimodal image description models for targeted prediction, the prediction accuracy can be improved.

[0070] Specifically, a training set corresponding to a plurality of feature combinations is obtained, wherein the training set corresponding to the feature combination includes a plurality of training samples and corresponding labels, and each training sample includes the feature value of each feature in the feature combination. According to the training sets corresponding to the plurality of feature combinations, the corresponding plurality of multimodal image description models are trained respectively to obtain each multimodal image description model corresponding to each feature combination. For example, if the feature combination is an image feature and a target time coding feature, a training set including a plurality of training samples is obtained, each training sample includes an image feature and a target time coding feature, and a multimodal image description model is trained according to the training set.

[0071] Furthermore, when there are multiple images to be processed, image recognition results of the multiple images to be processed are obtained; and based on the shooting time information of the multiple images to be processed, the multiple images to be processed and the corresponding image recognition results are processed into log records.

[0072] Among them, log records can be photo blogs or video blogs. Photo blogs (photo-log, plog) are mainly pictures, and pictures made by yourself or collected are placed in the blog according to time, type or other certain order. This kind of blog with pictures as the main focus is called a photo blog. Video blogs (video blog, vlog) are a type of blog, which means video records, video blogs, video network logs, emphasizing timeliness. Vlog authors use images instead of text or photos to write personal blogs and upload them to share with netizens.

[0073] Specifically, according to the sequence of shooting time of the multiple images to be processed, the multiple images to be processed and the corresponding description information are sequentially put into the blog template to generate a picture blog.

[0074] The multimodal features passed to the model in this application are in a unified format and do not contain redundant information, which effectively improves the efficiency of the model in utilizing external knowledge and speeds up calculations.

[0075] For further information, please refer to Figure 3 , Figure 3 is a flow chart of another embodiment of the image recognition method provided in the embodiment of the present application. Figure 3 As shown, the process of the image recognition method is as follows:

[0076] 301. Obtain an image to be processed, shooting time information of the image to be processed, and shooting location information of the image to be processed.

[0077] For example, the shooting time information is 13:14 on January 1, 22; the shooting location information is CC Street, BB District, AA City.

[0078] In a specific embodiment, obtaining the image to be processed, the shooting time information of the image to be processed, and the shooting location information of the image to be processed includes:

[0079] (1) Obtain the image to be processed.

[0080] In the embodiment of the present application, an image to be processed uploaded by a user is obtained.

[0081] (2) Obtain the EXIF ​​metadata of the image to be processed.

[0082] Specifically, before obtaining the metadata of the image to be processed, the metadata corresponding to the image and the picture obtained by taking the photo is stored in the album when taking the photo. The metadata corresponding to the picture includes the shooting time and shooting location. Among them, the shooting location can be obtained through the Global Positioning System (GPS) or the Beidou system. The longitude and latitude information of the image to be processed when it was taken is obtained through the Global Positioning System (GPS) or the Beidou system, and the administrative division address location when the image to be processed was taken is determined according to the longitude and latitude information. The format is like "AA City BB District CC Street" or "AA City BB District DD Famous Landmark".

[0083] (3) Extract the shooting time information and shooting location information of the image to be processed from the EXIF ​​metadata of the image to be processed.

[0084] 302. Perform feature extraction on the image to be processed to obtain image features.

[0085] Specifically, the CLIP model is used to extract features of the image to be processed to obtain image features in a preset two-dimensional array format. For example, the preset two-dimensional array format is a two-dimensional array of [x, m], where x represents the number of dimensions and m represents the feature value corresponding to the dimension.

[0086] 303. Obtain a preset number of time fields in the shooting time information.

[0087] The preset number t can be set according to specific circumstances, and the time fields of the preset number t can be set according to specific circumstances. For example, the preset number t is 5, and the fields of the preset number t are year, month, day, hour, and minute, respectively. The shooting time information is 13:14:15 on January 1, 22. The preset number of time fields in the shooting time information are obtained, and the 5 time fields are 22, 1, 1, 13, and 14, respectively.

[0088] 304. Input a preset number of time fields in the shooting time information into a preset convolutional neural network for encoding to obtain target time coding features.

[0089] The target time encoding feature includes a preset number of dimensions.

[0090] The preset convolutional neural network can be a neural network such as RNN, CNN, transformer, etc., which is not limited here. For example, the preset number t is 5, and the fields of the preset number t are year, month, day, hour and minute respectively. The shooting time information is 13:14:15 on January 1, 22. The preset number of time fields in the shooting time information is obtained, and the 5 time fields are 22, 1, 1, 13, and 14 respectively. The 5 time fields are input into the preset convolutional neural network for encoding to obtain the target time coding feature. The target time coding feature includes 5 dimensions, and the characteristic values ​​of the 5 dimensions of the target time coding feature are [22, 01, 01, 13, 14] respectively.

[0091] 305. Encode the shooting position information according to a preset character set to obtain an initial position coding feature.

[0092] The preset character set may be a character set of Unicode, which is also called the Universal Code or Single Code. It is developed by the Unicode Consortium and is an industry standard in the field of computer science, including character sets, encoding schemes, etc. Of course, the preset character set may also be an encoding using the American Standard Code for Information Interchange (ASCII) character set or Arabic numerals and Chinese characters.

[0093] 306. Input the initial position coding feature into a preset convolutional neural network for encoding to obtain the target position coding feature.

[0094] For example, the shooting location information is DD building on CC street in BB district in AA city, and the target location coding feature is obtained as [AA, BB, CC, DD], and the target location coding feature includes 4-dimensional features.

[0095] 307. The image features, the target time coding features, and the target position coding features are concatenated in a preset order to obtain a multimodal feature.

[0096] Specifically, the preset arrangement order is to arrange the image features, target time coding features and target position coding features in sequence. For example, if the image features are a two-dimensional array of [x, m], the dimension of the target time coding features is t, and the dimension of the target position coding features is l, then the image features, target time coding features and target position coding features are concatenated according to the preset arrangement order to obtain a multimodal feature, and the multimodal feature is a two-dimensional array of [x+t+l, m]. For example, x is 50 and m is 512.

[0097] 308. Determine a corresponding target multimodal image description model based on a feature combination in the multimodal features.

[0098] Among them, different feature combinations correspond to different multimodal image description models.

[0099] Specifically, the feature dimension of the multimodal feature is obtained, the feature combination in the multimodal feature is determined based on the feature dimension of the multimodal feature, and the corresponding target multimodal image description model is determined based on the feature combination in the multimodal feature. For example, the multimodal feature is a two-dimensional array of [x+t+l, m]. If the feature dimension of the multimodal feature is x+t+l, it indicates that the feature combination of the multimodal feature is image feature, target time coding feature and target time coding feature; if the feature dimension of the multimodal feature is x+t, it indicates that the feature combination of the multimodal feature is image feature and target time coding feature; if the feature dimension of the multimodal feature is x+l, it indicates that the feature combination of the multimodal feature is image feature and target time coding feature. Since the format of the multimodal feature of the present application is fixed, according to the multimodal feature.

[0100] 309. Input the multimodal features into the target multimodal image description model to obtain an image recognition result of the image to be processed.

[0101] In a specific embodiment, determining the image recognition result of the image to be processed based on the multimodal features includes: determining the corresponding target multimodal image description model based on the feature combination in the multimodal features, wherein different feature combinations correspond to different multimodal image description models; and inputting the multimodal features into the target multimodal image description model to obtain the image recognition result of the image to be processed. Since different feature combinations use different multimodal image description models for targeted prediction, the prediction accuracy can be improved.

[0102] Specifically, a training set corresponding to a plurality of feature combinations is obtained, wherein the training set corresponding to the feature combination includes a plurality of training samples and corresponding labels, and each training sample includes the feature value of each feature in the feature combination. The corresponding multimodal image description models are trained respectively according to the training sets corresponding to the plurality of feature combinations to obtain the multimodal image description models corresponding to the respective feature combinations. For example, if the feature combination is an image feature and a target time coding feature, a training set including a plurality of training samples is obtained, each training sample includes an image feature and a target time coding feature, and the multimodal image description model is trained according to the training set.

[0103] 310. Obtain image recognition results of multiple images to be processed.

[0104] When there are multiple images to be processed, image recognition results of the multiple images to be processed are obtained.

[0105] 311. Process the multiple images to be processed and the corresponding image recognition results into log records based on the shooting time information of the multiple images to be processed.

[0106] Among them, log records can be photo blogs or video blogs. Photo blogs (photo-log, plog) are mainly pictures, and pictures made by yourself or collected are placed in the blog according to time, type or other certain order. This kind of blog with pictures as the main focus is called a photo blog. Video blogs (video blog, vlog) are a type of blog, which means video records, video blogs, video network logs, emphasizing timeliness. Vlog authors use images instead of text or photos to write personal blogs and upload them to share with netizens.

[0107] Specifically, according to the sequence of shooting time of the multiple images to be processed, the multiple images to be processed and the corresponding description information are sequentially put into the blog template to generate a picture blog.

[0108] In order to facilitate better implementation of the image recognition method provided in the embodiment of the present application, the embodiment of the present application also provides an image recognition device based on the above image recognition method. The meanings of the terms are the same as those in the above image recognition method. For specific implementation details, please refer to the description in the above method embodiment.

[0109] Please refer to Figure 4 , Figure 4 The schematic diagram of the structure of the image recognition device provided in the embodiment of the present application is as follows. The image recognition device may include an acquisition module 401, a feature extraction module 402, an encoding module 403, a fusion module 404, and a recognition module 405, wherein:

[0110] The acquisition module 401 is used to acquire the image to be processed and the shooting information of the image to be processed;

[0111] A feature extraction module 402 is used to extract features from the image to be processed to obtain image features;

[0112] The encoding module 403 is used to encode the shooting information to obtain shooting features;

[0113] A fusion module 404 is used to fuse image features and shooting features to obtain multimodal features;

[0114] The recognition module 405 is used to determine the image recognition result of the image to be processed based on the multimodal features.

[0115] Obtaining the image to be processed and the shooting information of the image to be processed;

[0116] Perform feature extraction on the image to be processed to obtain image features;

[0117] Encode the shooting information to obtain shooting features;

[0118] Fusion of image features and shooting features to obtain multimodal features;

[0119] An image recognition result of the image to be processed is determined based on the multimodal features.

[0120] In an optional embodiment, the shooting information includes shooting position information and / or shooting time information of the image to be processed, and the shooting feature includes a target time coding feature and / or a target position coding feature; the fusion module 404 is used to:

[0121] The image features, target time coding features and / or target position coding features are spliced ​​in a preset arrangement order to obtain multimodal features.

[0122] In an optional embodiment, the shooting information includes shooting time information of the image to be processed, and the shooting feature includes a target time coding feature; the coding module 403 is used to:

[0123] Get a preset number of time fields in the shooting time information;

[0124] A preset number of time fields in the shooting time information are input into a preset convolutional neural network for encoding to obtain a target time coding feature, where the target time coding feature includes a preset number of dimensions.

[0125] In an optional embodiment, the shooting information includes shooting location information of the image to be processed, and the shooting feature includes a target location coding feature; the coding module 403 is used to:

[0126] Encode the shooting position information according to a preset character set to obtain an initial position coding feature;

[0127] The initial position encoding features are input into the preset convolutional neural network for encoding to obtain the target position encoding features.

[0128] In an optional embodiment, the acquisition module 401 is used to:

[0129] Obtaining an image to be processed;

[0130] Get the metadata of the image to be processed;

[0131] The shooting information of the image to be processed is extracted from the metadata of the image to be processed.

[0132] In an optional embodiment, determining an image recognition result of an image to be processed based on multimodal features includes:

[0133] Determining a corresponding target multimodal image description model based on a feature combination in the multimodal features, wherein different feature combinations correspond to different multimodal image description models;

[0134] The multimodal features are input into the target multimodal image description model to obtain the image recognition result of the image to be processed.

[0135] In an optional embodiment, the identification module 405 is used to:

[0136] Obtaining image recognition results of multiple images to be processed;

[0137] The plurality of images to be processed and the corresponding image recognition results are processed into log records based on the shooting time information of the plurality of images to be processed.

[0138] The specific implementation of each of the above modules can be found in the previous embodiments and will not be described in detail here.

[0139] Please refer to Figure 5 , Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0140] The electronic device may include components such as a processor 101 with one or more processing cores, a memory 102 with one or more computer-readable storage media, a power supply 103, and an input unit 104. Those skilled in the art will appreciate that the electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0141] The processor 101 is the control center of the electronic device, which uses various interfaces and lines to connect various parts of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 102, and calling data stored in the memory 102. Optionally, the processor 101 may include one or more processing cores; optionally, the processor 101 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 101.

[0142] The memory 102 can be used to store software programs and modules. The processor 101 executes various functional applications and data processing by running the software programs and modules stored in the memory 102. The memory 102 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 102 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 102 may also include a memory controller to provide the processor 101 with access to the memory 102.

[0143] The electronic device also includes a power supply 103 for supplying power to each component. Optionally, the power supply 103 can be logically connected to the processor 101 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 103 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0144] The electronic device may further include an input unit 104, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0145] Although not shown, the electronic device may also include a display unit, an image acquisition component, etc., which will not be described in detail here. Specifically in this embodiment, the processor 101 in the electronic device will load the executable code corresponding to one or more computer programs into the memory 102 according to the following instructions, and the processor 101 will execute the steps in the image recognition method provided by the present application, such as:

[0146] Obtaining the image to be processed and the shooting information of the image to be processed;

[0147] Perform feature extraction on the image to be processed to obtain image features;

[0148] Encode the shooting information to obtain shooting features;

[0149] Fusion of image features and shooting features to obtain multimodal features;

[0150] An image recognition result of the image to be processed is determined based on the multimodal features.

[0151] It should be noted that the electronic device provided in the embodiment of the present application and the image recognition method in the above embodiment belong to the same concept, and its specific implementation process is detailed in the above related embodiments and will not be repeated here.

[0152] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program stored therein is executed on a processor of an electronic device provided in an embodiment of the present application, the processor of the electronic device executes the steps in the image recognition method provided in the present application. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0153] The present application also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes various optional implementations of the above-mentioned image recognition method.

[0154] The above is a detailed introduction to an image recognition method and device provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0155] It should be noted that when the above embodiments of the present application are applied to specific products or technologies, the relevant data of the user is involved, and the user's permission or consent is required, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

Claims

1. An image recognition method, characterized in that: include: Acquire an image to be processed and shooting information of the image to be processed; Extracting features of the image to be processed to obtain image features; Encoding the shooting information to obtain shooting features; fusing the image feature and the shooting feature to obtain a multimodal feature; An image recognition result of the image to be processed is determined based on the multimodal features.

2. The image recognition method according to claim 1, characterized in that: The shooting information includes shooting position information and / or shooting time information of the image to be processed, and the shooting feature includes target time coding feature and / or target position coding feature; the fusing the image feature and the shooting feature to obtain the multimodal feature includes: The image features, the target time coding features and / or the target position coding features are spliced ​​in a preset arrangement order to obtain the multimodal features.

3. The image recognition method according to claim 1, characterized in that: The shooting information includes shooting time information of the image to be processed, and the shooting feature includes target time coding feature; encoding the shooting information to obtain the shooting feature includes: Obtain a preset number of time fields in the shooting time information; A preset number of time fields in the shooting time information are input into a preset convolutional neural network for encoding to obtain the target time coding feature, where the target time coding feature includes a preset number of dimensions.

4. The image recognition method according to claim 1, characterized in that: The shooting information includes shooting position information of the image to be processed, and the shooting feature includes target position coding feature; encoding the shooting information to obtain the shooting feature includes: Encoding the shooting position information according to a preset character set to obtain an initial position coding feature; The initial position coding feature is input into a preset convolutional neural network for encoding to obtain the target position coding feature.

5. The image recognition method according to claim 1, characterized in that: The step of obtaining the image to be processed and the shooting information of the image to be processed includes: Obtaining an image to be processed; Acquire metadata of the image to be processed; The shooting information of the image to be processed is extracted from the metadata of the image to be processed.

6. The image recognition method according to claim 1, characterized in that: The determining the image recognition result of the image to be processed based on the multimodal feature includes: Determining a corresponding target multimodal image description model based on a feature combination in the multimodal features, wherein different feature combinations correspond to different multimodal image description models; The multimodal features are input into the target multimodal image description model to obtain an image recognition result of the image to be processed.

7. The image recognition method according to claim 1, characterized in that: The determining the image recognition result of the image to be processed based on the multimodal feature includes: Obtaining image recognition results of a plurality of the images to be processed; The plurality of images to be processed and the corresponding image recognition results are processed into log records based on the shooting time information of the plurality of images to be processed.

8. An image recognition device, characterized in that: include: An acquisition module, used for acquiring an image to be processed and shooting information of the image to be processed; A feature extraction module is used to extract features from the image to be processed to obtain image features; An encoding module, used for encoding the shooting information to obtain shooting features; A fusion module, used for fusing the image feature and the shooting feature to obtain a multimodal feature; A recognition module is used to determine the image recognition result of the image to be processed based on the multimodal features.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor is used to run the computer program in the memory to execute the steps in the image recognition method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the image recognition method according to any one of claims 1 to 8.

11. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the image recognition method according to any one of claims 1 to 8 are implemented.

Images