Image determination method, device, storage medium and electronic device

Abstract

The embodiment of the present invention provides a method, device, storage medium and electronic device for determining an image, wherein the method comprises: using N sub-feature extraction networks in a trained target network model to extract features from an image containing a target object to obtain N sub-features of the target object; wherein different sub-feature extraction networks in the N sub-feature extraction networks are used to extract features for different postures of the object, and N is an integer greater than 1; fusing the N sub-features to obtain a target feature vector; based on the target feature vector, determining from the candidate images a candidate image whose image similarity with the target object is greater than an image similarity threshold. Through the present invention, the problem of inaccuracy in determining an image through an image in the related art is solved, and the accuracy of determining an image through an image is improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of communications, and in particular, to a method, device, storage medium and electronic device for determining an image. Background Art

[0002] In the related art, searching for images similar to the image in an image library through an image is increasingly widely used. The following uses the image of a non-motor vehicle as an example to illustrate the search image:

[0003] By inputting a picture of a non-motor vehicle, and then retrieving the non-motor vehicle images that appear in different cameras or videos, it has very important application value in urban security, public security criminal investigation, smart communities, smart e-commerce and other fields. However, in actual monitoring scenarios, due to the complex posture changes of non-motor vehicles, the semantic features between postures are not as highly reasonable as those of pedestrians, and the occlusion caused by the human body on the non-motor vehicle, the apparent features of the same non-motor vehicle may be greatly different, resulting in inaccurate images.

[0004] It can be seen from this that the related art has the problem of inaccuracy in determining the image through the image.

[0005] Currently, no effective solution has been proposed for the above-mentioned problems existing in the related technologies. Summary of the invention

[0006] The embodiments of the present invention provide a method, device, storage medium and electronic device for determining an image, so as to at least solve the problem of inaccurate image determination through an image in the related art.

[0007] According to one embodiment of the present invention, a method for determining an image is provided, comprising: using N sub-feature extraction networks in a trained target network model to perform feature extraction on an image containing a target object to obtain N sub-features of the target object, wherein different sub-feature extraction networks in the N sub-feature extraction networks are used to perform feature extraction for different postures of the object, and N is an integer greater than 1; fusing the N sub-features to obtain a target feature vector; and based on the target feature vector, determining from candidate images a candidate image whose image similarity with the target object is greater than an image similarity threshold.

[0008] According to another embodiment of the present invention, there is provided an image determination device, comprising: an extraction module, used to use N sub-feature extraction networks in a trained target network model to perform feature extraction on an image containing a target object, and obtain N sub-features of the target object, wherein different sub-feature extraction networks in the N sub-feature extraction networks are used to perform feature extraction for different postures of the object, and N is an integer greater than 1; a processing module, used to perform fusion processing on the N sub-features to obtain a target feature vector; and a determination module, used to determine, from candidate images, a candidate image whose image similarity with the target object is greater than an image similarity threshold, based on the target feature vector.

[0009] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, wherein the computer program implements the steps of any of the above methods when executed by a processor.

[0010] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0011] Through the present invention, N sub-feature extraction networks in the trained target network model are used to extract features from an image containing a target object to obtain N sub-features of the target object, and the N sub-feature vectors are fused to obtain a target feature vector, and a candidate image whose image similarity with the target object is greater than an image similarity threshold is determined from the candidate image according to the target feature vector. Since different sub-feature extraction networks in the N sub-feature extraction networks perform feature extraction for different postures of the object, the trained target network model can accurately determine the sub-features of the target object in the image under different postures, and then the sub-features under different postures are fused to obtain an accurate target feature vector, so that the target image can be accurately determined according to the target feature vector. Therefore, the problem of inaccuracy in determining an image through an image in the related art can be solved, and the accuracy of determining an image through an image is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a hardware structure block diagram of a mobile terminal of an image determination method according to an embodiment of the present invention;

[0013] Figure 2 is a flowchart of a method for determining an image according to an embodiment of the present invention;

[0014] Figure 3 is a target network model structure diagram according to an exemplary embodiment of the present invention;

[0015] Figure 4 is a flow chart of a method for determining an image according to a specific embodiment of the present invention;

[0016] Figure 5 4 is a structural block diagram of an image determination device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and in combination with the embodiments.

[0018] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0019] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a mobile terminal of an image determination method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0020] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for determining an image in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0021] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0022] In this embodiment, a method for determining an image is provided. Figure 2 is a flowchart of a method for determining an image according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0023] Step S202, using the N sub-feature extraction networks in the trained target network model to perform feature extraction on an image containing a target object to obtain N sub-features of the target object; wherein different sub-feature extraction networks in the N sub-feature extraction networks are used to perform feature extraction for different postures of the object, and N is an integer greater than 1;

[0024] Step S204, fusing the N sub-feature vectors to obtain a target feature vector;

[0025] Step S206: Based on the target feature vector, determine from the candidate images the candidate images whose image similarity with the target object is greater than an image similarity threshold.

[0026] In the above embodiment, the target object may be a non-motor vehicle, such as a motorcycle, an electric vehicle, a bicycle, a balance vehicle, etc. An image containing the target object may be input into the target network model, and the image may be feature extracted by the target network model to obtain N sub-features of the target object in different postures. Wherein, N may be 3, or may be other positive numbers greater than 1, and the present invention is not limited thereto. When N is 3, different postures may include the front, side, and back. That is, after inputting an image containing the target object, the front sub-feature vector, the side sub-feature, and the back sub-feature of the target object may be extracted from the image (extracting sub-features is the process of converting an image into an n-dimensional vector using a deep learning algorithm, and this n-dimensional vector contains the semantic information of the image). After extracting the sub-features of the target object in different postures, the sub-features in different postures may be fused to determine the target feature vector. According to the target feature vector, a candidate image whose image similarity with the target object is greater than the image similarity threshold is determined from the candidate image, so as to achieve the purpose of searching the image by image. Wherein, the image similarity threshold may be a pre-set threshold, and the image similarity threshold may be set to different thresholds according to the application scenario, and the present invention is not limited thereto.

[0027] Optionally, the executor of the above steps may be a background processor, or other devices with similar processing capabilities, or a machine that integrates at least an image acquisition device and a data processing device, wherein the image acquisition device may include a graphics acquisition module such as a camera, and the data processing device may include a computer, a mobile phone or other terminal, but is not limited thereto.

[0028] Through the present invention, N sub-feature extraction networks in the trained target network model are used to extract features from an image containing a target object to obtain N sub-features of the target object, and the N sub-feature vectors are fused to obtain a target feature vector, and a candidate image whose image similarity with the target object is greater than an image similarity threshold is determined from the candidate image according to the target feature vector. Since different sub-feature extraction networks in the N sub-feature extraction networks perform feature extraction for different postures of the object, the trained target network model can accurately determine the sub-features of the target object in the image under different postures, and then the sub-features under different postures are fused to obtain an accurate target feature vector, so that the target image can be accurately determined according to the target feature vector. Therefore, the problem of inaccuracy in determining an image through an image in the related art can be solved, and the accuracy of determining an image through an image is improved.

[0029] In an exemplary embodiment, the trained target network model is obtained by the following method: obtaining multiple groups of image samples containing historical objects; wherein different groups of image samples in the multiple groups of image samples include image samples collected from different angles for the historical objects, and the different angles correspond to the different postures; performing loss determination operations on the N sub-feature extraction networks respectively to obtain the predicted loss values ​​corresponding to each sub-feature extraction network in the N sub-feature extraction networks; the loss determination operation includes: inputting a group of image samples in the multiple groups of image samples into the target sub-feature extraction network, and determining the predicted loss value corresponding to the target sub-feature extraction network based on the feature extraction results of the target sub-feature extraction network for the group of image samples; the target sub-feature extraction network is the sub-feature extraction network corresponding to the group of image samples; based on the predicted loss values ​​corresponding to each sub-feature extraction network, adjusting the parameters of each sub-feature extraction network to obtain the trained target network model. In this embodiment, the target network model can be obtained through model training. When performing model training, multiple groups of image samples containing historical objects can be obtained, wherein each image sample in the multiple groups of image samples includes image samples obtained by shooting the historical objects from different angles. The initial network model of the target network model is trained through each group of image samples to obtain the target network model.

[0030] In the above embodiment, the target network model includes N sub-feature extraction networks, and each sub-feature extraction network can be trained using multiple groups of image samples, and the corresponding prediction loss value of each sub-feature extraction network can be determined. When determining the prediction loss value, it can be determined by a loss determination operation. A group of image samples from multiple groups of image samples is input into the target sub-feature extraction network, and the target sub-feature extraction network can determine the feature extraction results for a group of image samples, and the prediction loss value corresponding to the target sub-feature extraction network can be determined based on the extraction results. Among them, the target sub-feature extraction network is a sub-feature extraction network among the N sub-feature extraction networks.

[0031] In the above embodiment, the posture features to be extracted by each sub-feature extraction network can be preset. For example, when N is 3, three sub-feature extraction networks can be predetermined to respectively extract the front sub-features, side sub-features, and back sub-features of the target object. The posture of each image sample in each group of image samples input can be marked. For example, each group of image samples includes 3 images, which are the front image, side image, and back image of the object. Therefore, during training, the front image is input into the sub-feature extraction network for extracting the front sub-features, the side image is input into the sub-feature extraction network for extracting the side sub-features, and the back image is input into the sub-feature extraction network for extracting the back sub-features.

[0032] In the above embodiment, after the image sample is input into the target sub-feature extraction network, the prediction loss value can be determined, and the parameters of each sub-feature extraction network can be adjusted according to the prediction loss value to obtain a trained target network model.

[0033] In an exemplary embodiment, based on the predicted loss values ​​corresponding to each sub-feature extraction network, the parameters of each sub-feature extraction network are adjusted, including: performing the following operations on each sub-feature extraction network: based on the predicted loss value corresponding to one of the sub-feature extraction networks, adjusting the parameters of the one sub-feature extraction network; or weighted fusion of the predicted loss values ​​corresponding to the sub-feature extraction networks to obtain a comprehensive predicted loss value, and adjusting the parameters of each sub-feature extraction network based on the comprehensive predicted loss value. In this embodiment, when adjusting the parameters of each sub-feature extraction network, the predicted loss value of the sub-feature extraction network can be used to directly adjust the parameters of the sub-feature extraction network, or the predicted loss values ​​of each sub-feature extraction network can be weighted fusion to obtain a comprehensive predicted loss value, and the parameters of each sub-feature network can be adjusted according to the comprehensive predicted loss value.

[0034] In the above embodiment, the weight corresponding to each sub-feature extraction network can be customized according to the application scenario. The weight corresponding to each sub-feature extraction network can be the probability determined by the classification layer included in the target network model, or it can be other values. The present invention is not limited to this.

[0035] In an exemplary embodiment, weighted fusion of the prediction loss values ​​corresponding to each sub-feature extraction network includes: using the target classification layer included in the target network model to respectively determine the probability that the image samples input into each sub-feature extraction network belong to the target sub-feature extraction network; determining the probability as the target weight of the sub-feature extraction network; and weighted fusion of the target weight and the prediction loss values ​​corresponding to each sub-feature extraction network. In this embodiment, the target network model also includes a target classification layer, and the target classification layer can be a processing layer parallel to the N sub-feature extraction networks. During the training process, each group of image samples included in the multiple groups of image samples is input into the initial model of the target network model, and the target classification layer can classify each image in each group of image samples to determine the probability that the image samples input into each sub-feature extraction network belong to the target sub-feature extraction network, and determine the target weight of the sub-feature extraction network based on this probability. Determine the product of each target weight and the prediction loss function determined by the sub-feature extraction network corresponding to the target weight, and determine the comprehensive prediction loss value by the sum of multiple products.

[0036] In the above embodiment, when the N sub-feature extraction networks are 3 sub-feature extraction networks, the losses generated by the three branch networks of front, side and back can be multiplied by the probability of the corresponding posture of the model in the S1 layer softmax output, and finally the losses of the three branches are added as the final loss of the network, so as to back-propagate the gradient and update the network parameters. During the training process, map can be used as a test indicator, and the model with the highest map in the test set can be saved as the target network model. Among them, map is an indicator for calculating the effect of image search. For the retrieval target, the higher the ranking of all its true values ​​in the retrieval results, the higher the map, and vice versa.

[0037] In an exemplary embodiment, each of the plurality of groups of image samples includes: a plurality of image samples obtained by photographing the historical object from different angles and label information of each of the image samples, wherein the label information includes identification information of the image sample and angle information of photographing the image sample. In this embodiment, when the target object is a non-motor vehicle, for the same non-motor vehicle, images of different postures in different monitoring scenes can be collected as training images, and the same non-motor vehicle is marked with its ID (i.e. identification information) and posture (i.e. angle information, such as front, side, back).

[0038] In an exemplary embodiment, inputting a group of image samples from the plurality of groups of image samples into the target sub-feature extraction network includes: weakening the target area of ​​the group of image samples to obtain a first image sample; enhancing the first image sample to obtain a second image sample; and inputting the second image sample into the target sub-feature extraction network. In this embodiment, when the target object is a non-motor vehicle, when a person is driving a non-motor vehicle, the legs usually block the body of the non-motor vehicle. Therefore, weakening the target area of ​​the image sample can reduce the effect of the human body on the body occlusion. When the target area is weakened, during the training process, the model will implicitly consider the target area to be an unimportant area, and will suppress the human body information in the target area to a certain extent.

[0039] In the above embodiment, after the training image is weakened to obtain the first image sample, the first image sample can be enhanced to obtain the second image sample. The enhancement operation may include image enhancement operations such as random cropping, random erasing, and color jittering. The second image sample is then used to train the target sub-feature extraction network to obtain the target network model.

[0040] In an exemplary embodiment, weakening the target area of ​​the group of image samples to obtain the first image includes: determining the target probability of weakening the target area; and setting the target area to zero according to the target probability to obtain the first image sample. In this embodiment, when the target object is a non-motor vehicle, the collected non-motor vehicle image contains a human body. In order to eliminate the interference of the human body on the non-motor vehicle, the upper half of the image included in each group of image samples can be set to zero for the target probability (for example, 0.5 probability, this value is only an exemplary description, and can also be 0.4 probability, 0.6 probability, etc., the present invention is not limited to this), that is, 0.5 probability makes the upper half of the image have no information. Among them, the target area can be determined according to the type of the target object. For example, when there is interference in the lower half of the image sample, the lower half of the image can be determined as the target area; when there is interference in the upper half of the image sample, the upper half of the image can be determined as the target area; when there is interference in the left half of the image sample, the left half of the image can be determined as the target area; when there is interference in the right half of the image sample, the right half of the image can be determined as the target area. The present invention does not limit the target area.

[0041] In the above embodiment, the image target area is set to 0 with a probability of 0.5. This operation is mainly to reduce the influence of the human body on the non-motor vehicle. The probability of 0.5 will make the model have a 50% chance of not receiving information from the upper half of the image during the training process. Therefore, during the training process, the model will implicitly think that the upper half is an unimportant area and will suppress the human body information in the upper half to a certain extent. Why not erase the upper half directly? Because in the real monitoring scene, the posture of non-motor vehicles and the viewing angle of the camera are relatively rich. If the upper half is erased directly, the important information of the non-motor vehicle may also be erased. Therefore, this method of setting the upper half of the area to 0 with a probability of 0.5 is conducive to reducing the interference of the human body on the non-motor vehicle on the non-motor vehicle retrieval task and improving the accuracy of non-motor vehicle retrieval.

[0042] In an exemplary embodiment, determining the prediction loss value corresponding to the target sub-feature extraction network based on the feature extraction result of the target sub-feature extraction network for the set of image samples includes: determining the first feature extraction image of the target sub-feature extraction network based on the feature extraction result; segmenting the first feature extraction image according to a predetermined direction to obtain a plurality of second feature extraction images of different angle categories; and determining the prediction loss value based on the second feature image. In this embodiment, the first feature extraction image can be determined by the feature extraction result of the target sub-feature extraction network, and then the first feature image is segmented according to a predetermined direction to obtain second feature images of different angle categories. Wherein, the predetermined direction can be determined by the target area, when the target area is the upper half or the lower half, the predetermined direction can be the left-right direction, and when the target area is the left half or the right half, the predetermined direction can be the up-down direction. For example, when the target object is a non-motor vehicle, the first feature image can be cut in half up and down according to the left-right direction, divided into two blocks of L1 and L2, and then only the lower half of the L2 feature is taken for subsequent SoftmaxLoss and TripletLoss calculations.

[0043] In the above embodiment, the feature map is divided into upper and lower blocks, and then only the lower block feature map is forward propagated to participate in the subsequent loss calculation, so that the feature information of the upper part does not contribute to the update of the network. Since the non-motor vehicle contains human bodies, there is a relative upper and lower position relationship between the human body and the non-motor vehicle. By prohibiting the upper part of the features from generating loss, the influence of the human body on the non-motor vehicle retrieval accuracy can be weakened to a certain extent, the interference of the human body on the non-motor vehicle on the non-motor vehicle semantic information can be weakened, more feature information of the non-motor vehicle can be extracted, and the calculation speed of the model can be improved.

[0044] In an exemplary embodiment, before using the N sub-feature extraction networks in the trained target network model to extract features from an image containing a target object, the method further includes: adding a target classification layer and the N sub-feature extraction networks in parallel after the last convolution layer of the residual network model to obtain an initial network model, wherein the target classification layer is used to divide the feature image output by the last convolution layer into the N posture categories, and each sub-feature extraction network is used to output a feature image corresponding to each sub-feature extraction network; and the initial network model is trained using the multiple groups of image samples to obtain the target network model. In this embodiment, before using the target network model, an initial network model may be first constructed and trained to obtain the target network model. The structure diagram of the target network model can be found in the attached figure. Figure 3 ,like Figure 3As shown, a target classification layer and N sub-feature extraction networks can be added in parallel after the last convolution layer of the residual network model to obtain an initial network model. Among them, the residual network model can be resnet50, and the target classification layer can be a softmax classification layer. That is, resnet50 can be used as the backbone network, and after the output of the last convolution layer of resnet50, a softmax classification layer S1 is connected to perform three classifications of the image, front, side and back. At the same time, three branch networks are connected after the last convolution layer of resnet50, which are used to extract the apparent features of the front, back and side of the non-motor vehicle respectively. Among them, the branch network can extract the apparent features of the target object according to the angle information included in the label information of the training image.

[0045] In the above embodiment, three branch networks are introduced into the model for extracting the front, side and back respectively. The introduction of this branch structure will enable the target network model to have the ability to extract the three posture information of the front, back and side of the target object at the same time; the introduction of this branch structure can also allow each branch to focus more on extracting the apparent information of the target object under a certain posture, so that the model can fit better; finally, the features of the three branches are fused to calculate the similarity. The fusion of the features of the three branches will greatly reduce the semantic differences between different postures of non-motor vehicles. The fused features will contain richer information, which can greatly improve the ability of target objects to search each other across postures and improve the accuracy of image search.

[0046] In an exemplary embodiment, the N sub-features are fused to obtain a target feature vector, including: splicing the N sub-features to obtain a spliced ​​feature vector; and determining the spliced ​​feature vector as the target feature vector. In this embodiment, a non-motor vehicle image to be retrieved can be input into the target network model, and the feature image can be determined using the target network model, that is, the feature vectors of the three branches of the front, side and back are extracted, and then the three feature vectors are spliced ​​together as the final target feature vector of the image. Then, the same feature extraction operation is performed on the non-motor vehicle base image, and then the cosine similarity between the feature vectors of the image to be retrieved and the non-motor vehicle base image is calculated. Finally, the similarity between the base image and the image to be retrieved is sorted from large to small, and the image search result of the non-motor vehicle is obtained. The non-motor vehicle with higher similarity indicates that it is more likely to be the same vehicle as the non-motor vehicle to be retrieved.

[0047] In an exemplary embodiment, based on the target feature vector, determining from the candidate images a candidate image whose image similarity with the target object is greater than an image similarity threshold includes: inputting the candidate image into the trained target network model, extracting candidate features of the candidate object included in the candidate image in different postures; fusing the candidate features to obtain a target candidate feature vector; and based on the target feature vector and the target candidate feature vector, determining from the candidate images a candidate image whose image similarity with the target object is greater than an image similarity threshold. In this embodiment, when determining the target image, the image in the candidate image can be input into the target network model to determine the candidate features of each image, and the candidate features are fused to obtain a target candidate feature vector. The target candidate feature vector is compared with the target feature vector to determine the similarity between the two, and the similarity is sorted. The top-ranked image is determined as the candidate image, or the image with a similarity greater than the image similarity threshold is determined as the candidate image.

[0048] The following describes the image determination method in conjunction with a specific implementation method:

[0049] Figure 4 is a flow chart of a method for determining an image according to a specific embodiment of the present invention. Figure 4 As shown, the process includes:

[0050] Step S402, inputting the non-motor vehicle image to be retrieved and the non-motor vehicle base image into the non-motor vehicle image search model (corresponding to the above-mentioned target network model);

[0051] Step S404, obtaining the front, side and back branch features of the non-motor vehicle;

[0052] Step S406, fusion of the front, side and back features of the non-motor vehicle;

[0053] Step S408, determining the features of the image to be retrieved and the features of the base image;

[0054] Step S410, similarity calculation;

[0055] Step S412, similarity sorting;

[0056] Step S414, determining the search results.

[0057] In the above-mentioned embodiment, the multi-posture information of non-motor vehicles is introduced in the model training stage, and three posture branches are constructed to improve the ability of non-motor vehicles to search each other across postures; by allowing the model to implicitly ignore the upper part of the human body during the training process and focusing on the non-motor vehicle in the lower part, the interference of human body information on the model is reduced, and the model's ability to express non-motor vehicles is improved. The problem that the different postures of non-motor vehicles are very different in the related technology, making it difficult to rank the pictures of the same non-motor vehicle in cross-posture mutual search, and the problem that in actual monitoring scenarios, there are human bodies on non-motor vehicles, and the information of the human body will interfere with the retrieval of non-motor vehicles.

[0058] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0059] In this embodiment, a device for determining an image is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0060] Figure 5 is a structural block diagram of an image determination device according to an embodiment of the present invention, such as Figure 5 As shown, the device comprises:

[0061] An extraction module 52 is used to use the N sub-feature extraction networks in the trained target network model to perform feature extraction on an image containing a target object to obtain N sub-features of the target object; wherein different sub-feature extraction networks in the N sub-feature extraction networks are used to perform feature extraction for different postures of the object, and N is an integer greater than 1;

[0062] A processing module 54 is used to perform fusion processing on the N sub-features to obtain a target feature vector;

[0063] The determination module 56 is configured to determine, based on the target feature vector, from the candidate images, a candidate image whose image similarity with the target object is greater than an image similarity threshold.

[0064] In an exemplary embodiment, the device can obtain the trained target network model in the following manner: obtain multiple groups of image samples containing historical objects; wherein different groups of image samples in the multiple groups of image samples include image samples collected from different angles for the historical objects, and the different angles correspond to the different postures; for the N sub-feature extraction networks, respectively, a loss determination operation is performed to obtain a predicted loss value corresponding to each sub-feature extraction network in the N sub-feature extraction networks; the loss determination operation includes: inputting a group of image samples from the multiple groups of image samples into the target sub-feature extraction network, and determining the predicted loss value corresponding to the target sub-feature extraction network based on the feature extraction results of the target sub-feature extraction network for the group of image samples; the target sub-feature extraction network is the sub-feature extraction network corresponding to the group of image samples; based on the predicted loss values ​​corresponding to each sub-feature extraction network, parameters of each sub-feature extraction network are adjusted to obtain the trained target network model.

[0065] In an exemplary embodiment, the device can implement parameter adjustment for each sub-feature extraction network based on the predicted loss value corresponding to each sub-feature extraction network in the following manner: performing the following operations on each sub-feature extraction network respectively: adjusting the parameters of one sub-feature extraction network based on the predicted loss value corresponding to the one sub-feature extraction network; or performing weighted fusion on the predicted loss values ​​corresponding to the sub-feature extraction networks to obtain a comprehensive predicted loss value, and adjusting the parameters of each sub-feature extraction network based on the comprehensive predicted loss value.

[0066] In an exemplary embodiment, the device can achieve weighted fusion of the prediction loss values ​​corresponding to the various sub-feature extraction networks in the following manner: using the target classification layer included in the target network model to respectively determine the probability that the image samples input into the various sub-feature extraction networks belong to the target sub-feature extraction network; determining the probability as the target weight of the sub-feature extraction network; and performing weighted fusion of the target weight and the prediction loss values ​​corresponding to the various sub-feature extraction networks.

[0067] In an exemplary embodiment, each group of image samples in the multiple groups of image samples includes: multiple image samples obtained by photographing historical objects from different angles and label information of each of the image samples, wherein the label information includes identification information of the image sample and angle information of photographing the image sample.

[0068] In an exemplary embodiment, the device can input a group of image samples from the multiple groups of image samples into the target sub-feature extraction network in the following manner: weaken the target area of ​​the group of image samples to obtain a first image sample; enhance the first image sample to obtain a second image sample; and input the second image sample into the target sub-feature extraction network.

[0069] In an exemplary embodiment, the device can implement weakening processing on the target area of ​​the group of image samples to obtain the first image sample in the following manner: determining the target probability of weakening the target area; and performing zero processing on the target area according to the target probability to obtain the first image sample.

[0070] In an exemplary embodiment, the device can determine the predicted loss value corresponding to the target sub-feature extraction network based on the feature extraction result of the target sub-feature extraction network for the group of image samples in the following manner: determine the first feature extraction image of the target sub-feature extraction network based on the feature extraction result; segment the first feature extraction image according to a predetermined direction to obtain a second feature extraction image of multiple different angle categories; and determine the predicted loss value based on the second feature extraction image.

[0071] In an exemplary embodiment, the device can be used to analyze an image containing a target object using a target network model to extract feature vectors of the target object at different angles, and then add a target classification layer and the N sub-feature extraction networks in parallel after the last convolution layer of the residual network model to obtain an initial network model, wherein the target classification layer is used to divide the feature image output by the last convolution layer into N posture categories, and each of the N sub-feature extraction networks is used to output a feature image corresponding to each of the N sub-feature extraction networks; the initial network model is trained using the multiple groups of image samples to obtain the target network model.

[0072] In an exemplary embodiment, the processing module 54 may implement fusion processing on the N sub-features to obtain a target feature vector in the following manner: concatenate the N sub-features to obtain a concatenated feature vector; and determine the concatenated feature vector as the target feature vector.

[0073] In an exemplary embodiment, the determination module 56 can be implemented in the following manner to determine, from the candidate images, a candidate image whose image similarity with the target object is greater than an image similarity threshold based on the target feature vector: input the candidate image into the target network model, and extract candidate features of the candidate object in different postures included in the candidate image; fuse the candidate features to obtain a target candidate feature vector; and determine, from the candidate images, a candidate image whose image similarity with the target object is greater than an image similarity threshold based on the target feature vector and the target candidate feature vector.

[0074] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0075] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0076] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0077] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0078] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0079] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0080] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0081] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for determining an image, characterized in that: include: Using the N sub-feature extraction networks in the trained target network model, feature extraction is performed on an image containing a target object to obtain N sub-features of the target object; wherein different sub-feature extraction networks in the N sub-feature extraction networks are used to extract features for different postures of the object, and N is an integer greater than 1; The N sub-features are fused to obtain a target feature vector; Based on the target feature vector, determining from the candidate images a candidate image whose image similarity with the target object is greater than an image similarity threshold; The target network model is obtained by inputting a group of image samples from multiple groups of image samples into a target sub-feature extraction network for training, wherein the target sub-feature extraction network is a sub-feature extraction network corresponding to the group of image samples; inputting a group of image samples from the multiple groups of image samples into the target sub-feature extraction network includes: weakening the target area of ​​the group of image samples to obtain a first image sample; enhancing the first image sample to obtain a second image sample; inputting the second image sample into the target sub-feature extraction network; weakening the target area of ​​the group of image samples to obtain a first image sample includes: determining a target probability of weakening the target area; and zeroing the target area according to the target probability to obtain the first image sample.

2. The method according to claim 1, characterized in that The trained target network model is obtained in the following way: Acquire multiple groups of image samples containing historical objects; wherein different groups of image samples in the multiple groups of image samples include image samples collected from different angles for the historical objects, and the different angles correspond to the different postures; For the N sub-feature extraction networks, loss determination operations are performed respectively to obtain predicted loss values ​​corresponding to each sub-feature extraction network in the N sub-feature extraction networks; the loss determination operation includes: inputting a group of image samples from the multiple groups of image samples into a target sub-feature extraction network, and determining the predicted loss value corresponding to the target sub-feature extraction network based on the feature extraction results of the target sub-feature extraction network for the group of image samples; the target sub-feature extraction network is a sub-feature extraction network corresponding to the group of image samples; Based on the prediction loss values ​​corresponding to the sub-feature extraction networks, parameters of the sub-feature extraction networks are adjusted to obtain the trained target network model.

3. The method according to claim 2, characterized in that Based on the prediction loss values ​​corresponding to the sub-feature extraction networks, parameters of the sub-feature extraction networks are adjusted, including: The following operations are performed for each of the sub-feature extraction networks: based on the prediction loss value corresponding to one of the sub-feature extraction networks, parameters of the sub-feature extraction network are adjusted; or The prediction loss values ​​corresponding to the various sub-feature extraction networks are weighted and fused to obtain a comprehensive prediction loss value, and parameters of the various sub-feature extraction networks are adjusted based on the comprehensive prediction loss value.

4. The method according to claim 3, characterized in that The weighted fusion of the prediction loss values ​​corresponding to each sub-feature extraction network includes: Determine the probability that the image samples input into each of the sub-feature extraction networks belong to the target sub-feature extraction network by using the target classification layer included in the target network model; Determining the probability as a target weight of the sub-feature extraction network; The target weight and the prediction loss values ​​corresponding to each sub-feature extraction network are weightedly fused.

5. The method according to claim 2, characterized in that: Each group of image samples in the multiple groups of image samples includes: A plurality of image samples obtained by photographing the historical object from different angles and label information of each of the image samples, wherein the label information includes identification information of the image sample and angle information of photographing the image sample.

6. The method according to claim 2, characterized in that Determining a prediction loss value corresponding to the target sub-feature extraction network based on a feature extraction result of the target sub-feature extraction network for the set of image samples includes: Determine a first feature extraction image of the target sub-feature extraction network based on the feature extraction result; Segmenting the first feature extraction image according to a predetermined direction to obtain a plurality of second feature extraction images of different angle categories; The predicted loss value is determined based on the second feature extraction image.

7. The method according to claim 1, characterized in that Before using the N sub-feature extraction networks in the trained target network model to extract features from an image containing a target object, the method further includes: Adding a target classification layer and the N sub-feature extraction networks in parallel after the last convolution layer of the residual network model to obtain an initial network model, wherein the target classification layer is used to classify the feature image output by the last convolution layer into N posture categories, and each sub-feature extraction network is used to output a feature image corresponding to each sub-feature extraction network; The initial network model is trained using multiple groups of image samples to obtain the target network model.

8. The method according to claim 1, characterized in that The N sub-features are fused to obtain a target feature vector including: The N sub-features are concatenated to obtain a concatenated feature vector; The concatenated feature vector is determined as the target feature vector.

9. The method according to claim 1, characterized in that: Based on the target feature vector, determining from the candidate images a candidate image whose image similarity with the target object is greater than an image similarity threshold comprises: Inputting the candidate image into the trained target network model to extract candidate features of the candidate object in different postures included in the candidate image; The candidate features are fused to obtain a target candidate feature vector; Based on the target feature vector and the target candidate feature vector, a candidate image having an image similarity with the target object greater than an image similarity threshold is determined from the candidate images.

10. An image determination device, characterized in that: include: An extraction module, used to use N sub-feature extraction networks in the trained target network model to perform feature extraction on an image containing a target object to obtain N sub-features of the target object; wherein different sub-feature extraction networks in the N sub-feature extraction networks are used to perform feature extraction for different postures of the object, and N is an integer greater than 1; A processing module, used for fusing the N sub-features to obtain a target feature vector; A determination module, for determining, based on the target feature vector, from the candidate images, a candidate image whose image similarity with the target object is greater than an image similarity threshold; The device is used to input a group of image samples from multiple groups of image samples into a target sub-feature extraction network for training to obtain a target network model; wherein the target sub-feature extraction network is a sub-feature extraction network corresponding to the group of image samples; the device implements inputting a group of image samples from the multiple groups of image samples into the target sub-feature extraction network in the following manner: weakening a target area of ​​the group of image samples to obtain a first image sample; enhancing the first image sample to obtain a second image sample; inputting the second image sample into the target sub-feature extraction network; the device implements weakening a target area of ​​the group of image samples to obtain a first image sample in the following manner: determining a target probability of weakening the target area; and zeroing the target area according to the target probability to obtain the first image sample.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 9 when executed by a processor.

12. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 9.

Images