Image Recognition Method, Apparatus, Electronic Device, and Storage Medium

Abstract

The present invention provides an image recognition method, apparatus, electronic device and storage medium. The method includes: determining an image to be recognized; extracting features of the image to be recognized based on a feature extraction network to obtain image features of the image to be recognized, where the feature extraction network is trained jointly with an initial semantic segmentation network based on a first sample image carrying a target category label and a target segmentation label, and the initial semantic segmentation network performs semantic segmentation based on the features extracted by the feature extraction network; determining a recognition result of the image to be recognized based on the image features. The method, apparatus, electronic device and storage medium provided by the present invention enable the feature extraction network to learn more feature information of the target region during the training process, reduce the mislearning of background information, improve the accuracy of feature extraction of the target region by the feature extraction network, and further improve the accuracy of classification and recognition of the target region.

Description

Technical Field

[0001] The present invention relates to the technical field of machine vision, and particularly to an image recognition method, device, electronic device and storage medium. Background Art

[0002] Whether employees wear work uniforms as required by the enterprise is very important for their own safety. In all walks of life, there are situations where on-the-job workers do not wear work uniforms for dangerous operations, resulting in unnecessary casualties and losses. Therefore, many enterprises have formulated relevant regulations on the wearing of work uniforms by their staff.

[0003] In order to supervise employees' wearing of work uniforms, work uniform recognition is mainly carried out through a feature detection model. However, due to a large amount of background information when taking pictures of workers, even if a rectangular frame is extracted for the target area of interest, there is still a large amount of background information, which causes the feature detection model to be prone to mislearning during the learning process, and thus the accuracy of work uniform recognition of the feature detection model is relatively low during testing. Summary of the Invention

[0004] The present invention provides an image recognition method, device, electronic device and storage medium to solve the defect that the accuracy of work uniform recognition of the feature detection model in the prior art is relatively low.

[0005] The present invention provides an image recognition method, including:

[0006] Determine the image to be recognized;

[0007] Based on a feature extraction network, extract features of the image to be recognized to obtain the image features of the image to be recognized. The feature extraction network is trained jointly with an initial semantic segmentation network based on a first sample image carrying a target category label and a target segmentation label, and the initial semantic segmentation network performs semantic segmentation based on the features extracted by the feature extraction network;

[0008] Based on the image features, determine the recognition result of the image to be recognized.

[0009] According to an image recognition method provided by the present invention, the training steps of the feature extraction network include:

[0010] Based on an initial feature extraction network, determine the initial features of the first sample image;

[0011] Based on the similarity between the initial features of the first sample images under the same target category, and / or the similarity between the initial features of the first sample images under different target categories, determine a first loss;

[0012] Based on the initial semantic segmentation network and the target segmentation label, determine a second loss;

[0013] Based on the first loss, perform parameter iteration on the initial feature extraction network, and based on the second loss, perform parameter iteration on the initial feature extraction network and the initial semantic segmentation network to obtain the feature extraction network.

[0014] According to an image recognition method provided by the present invention, determining the first loss based on the similarity between the initial features of the first sample images under the same target category and / or the similarity between the initial features of the first sample images under different target categories includes:

[0015] Based on the first sample images, determine multiple batches of sample images;

[0016] Based on the initial features of the sample images in the current batch and the corresponding target category labels of the sample images in the current batch, and the initial features of the sample images in the batches before the current batch and the corresponding target category labels of the sample images in the batches before the current batch, construct a first positive and negative sample pair and a second positive and negative sample pair. The two images in the first positive and negative sample pair both come from the current batch, and one sample image in the second positive and negative sample pair comes from the current batch and the other sample image comes from the batches before the current batch;

[0017] Based on the similarity of the first positive and negative sample pair and the second positive and negative sample pair, determine the first loss.

[0018] According to an image recognition method provided by the present invention, the step of constructing the second positive and negative sample pair includes:

[0019] Based on the similarity between the initial features of the sample images in the current batch and the similarity between the initial features of the sample images in the same batch in the batches before the current batch, determine the sampling ratio;

[0020] Based on the initial features of the sample images in the current batch and the initial features of the sample images in the batches before the current batch, and the sampling ratio, construct the second positive and negative sample pair.

[0021] According to an image recognition method provided by the present invention, determining the image to be recognized includes:

[0022] Obtain the original image, perform target area detection on the original image, and use the target area in the original image as the image to be recognized.

[0023] According to an image recognition method provided by the present invention, determining the recognition result of the image to be recognized based on the image features includes:

[0024] Determine the recognition result of the image to be recognized based on the similarity between the image features and the sample image features of the second sample image corresponding to the target category.

[0025] According to an image recognition method provided by the present invention, the second sample image includes images taken from different shooting angles under the same target category.

[0026] The present invention also provides an image recognition device, including:

[0027] A determination module, configured to determine an image to be recognized;

[0028] An extraction module, configured to extract features of the image to be recognized based on a feature extraction network, to obtain the image features of the image to be recognized, where the feature extraction network is trained jointly with an initial semantic segmentation network based on a first sample image carrying a target category label and a target segmentation label, and the initial semantic segmentation network performs semantic segmentation based on the features extracted by the feature extraction network;

[0029] A recognition module, configured to determine the recognition result of the image to be recognized based on the image features.

[0030] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the image recognition method as described in any one of the above when executing the program.

[0031] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the image recognition method as described in any one of the above.

[0032] The present invention also provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the image recognition method as described in any one of the above.

[0033] The image recognition method, device, electronic device, and storage medium provided by the present invention, by applying a feature extraction network trained jointly with an initial semantic segmentation network based on a first sample image carrying a target category label and a target segmentation label, extract features of the image to be recognized, and obtain a recognition result according to the extracted image features, realizing that the feature extraction network can learn more feature information of the target area during the training process, reducing the mislearning of background information, improving the accuracy of feature extraction of the target area by the feature extraction network, and further improving the accuracy of classification and recognition of the target area. Description of the Drawings

[0034] To more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can also be obtained based on these drawings.

[0035] Figure 1 It is a schematic flowchart of the image recognition method provided by the present invention;

[0036] Figure 2 It is a schematic flowchart of the feature extraction network training method provided by the present invention;

[0037] Figure 3 It is a schematic network structure diagram of the feature extraction network and the semantic segmentation network provided by the present invention;

[0038] Figure 4 It is a schematic flowchart of the first loss determination method provided by the present invention;

[0039] Figure 5 It is a schematic flowchart of the method for adaptively constructing the second positive and negative sample pairs provided by the embodiments of the present invention;

[0040] Figure 6 It is a schematic flowchart of the feature extraction network learning process provided by the present invention;

[0041] Figure 7 It is a schematic flowchart of the feature extraction network testing method provided by the present invention;

[0042] Figure 8 It is a schematic structure diagram of the image recognition device provided by the present invention;

[0043] Figure 9 It is a schematic structure diagram of the electronic device provided by the present invention. Detailed implementation manners

[0044] To make the objectives, technical solutions, and advantages of the present invention clearer, the following will clearly and completely describe the technical solutions in the present invention in conjunction with the accompanying drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments in the present invention belong to the scope of protection of the present invention.

[0045] Currently, due to a large amount of background information when capturing workers' images, even if rectangular frames are extracted for the target regions of interest, there is still a large amount of background information, which is likely to cause the feature detection model to mis-learn the background information during the feature learning process, thus unable to learn the work uniform texture information well, and further resulting in a low accuracy rate in work uniform recognition.

[0046] Therefore, how to improve the accuracy of image target region recognition in a complex background environment is a technical problem that needs to be urgently solved by those skilled in the art.

[0047] In view of the above technical problems, an embodiment of the present invention provides an image recognition method. Figure 1 It is a schematic flowchart of the image recognition method provided by the present invention. As Figure 1 shown, this method is mainly applied to the work uniform detection scenario. The following embodiments will be described and stated with the work uniform detection scenario as an example. In addition, this method can also be applied to similar scenarios such as personnel behavior detection. The method includes:

[0048] Step 110, determine the image to be recognized;

[0049] It should be noted that the image to be recognized can be a full-body or half-body image of a single person. The image to be recognized can be directly captured, or can be processed based on the captured image. For example, if there are multiple people in the captured image, each person needs to be segmented as a target into corresponding sub-images, and each sub-image needs to be recognized as the image to be recognized. The embodiments of the present invention do not limit this.

[0050] Step 120, based on the feature extraction network, extract features from the image to be recognized to obtain the image features of the image to be recognized. The feature extraction network is trained jointly with the initial semantic segmentation network based on the first sample image carrying the target category label and the target segmentation label. The initial semantic segmentation network performs semantic segmentation based on the features extracted by the feature extraction network;

[0051] Step 130, based on the image features, determine the recognition result of the image to be recognized.

[0052] In order to reduce the influence of the complex background in the image on the training of the feature extraction network, considering that the feature extraction network itself has the characteristics of changing the image from low-dimensional features to high-dimensional features and the feature size changing from large to small, this characteristic conforms to the downsampling part in the idea of first performing downsampling and then upsampling in semantic segmentation. Therefore, an embodiment of the present invention introduces the semantic segmentation network to be jointly trained with the feature extraction network, so that the feature extraction network can perceive the target region of semantic segmentation, and thus the focus of feature extraction is tilted to the target region, ignoring the influence of the complex background.

[0053] It can be understood that the target area here is the area in the image to be recognized where image recognition needs to be performed. The target of the semantic segmentation network jointly trained with the feature extraction network for segmentation is the area where image recognition needs to be performed. For example, in the work uniform detection scenario, the target area is the area of the clothing worn by the person in the image to be recognized.

[0054] Specifically, for the feature extraction network applied in step 120, the feature extraction network can be trained through the first sample image and the target class label of the first sample image. At the same time, through the first sample image and the target segmentation label of the first sample image, the feature extraction network can be assisted in training in conjunction with the initial semantic segmentation network. After the training is completed, the feature extraction network can be obtained. In this process, the feature extraction network can apply the idea of metric learning. Through the first sample image and the target class label of the first sample image, it can learn the differences in image features of the first sample images of different target classes and the commonalities in image features of the first sample images of the same target class, thereby enhancing the ability of the feature extraction network in image feature extraction. Moreover, through the joint training with the semantic segmentation network, the feature extraction network can better tilt the focus of image feature extraction to the target area, so that the extracted image features can more prominently represent the features of the target area, facilitating better image recognition in the subsequent process.

[0055] For example, in the work uniform detection scenario, the target class label in the first sample image applied in the training process of the feature extraction network is used to indicate the type of the work uniform worn by the person in the image, such as whether the person is wearing a work uniform, the category or color of the work uniform worn, etc. The target segmentation label is used to indicate the area of the work uniform worn by the person in the image. During the training process, the feature extraction network can learn the differences in image features of the first sample images of different work uniform types and the commonalities in image features of the first sample images of the same work uniform type. At the same time, it can tilt the focus of image feature extraction to the work uniform area, so that the extracted image features can more prominently represent the features of the work uniform area.

[0056] After obtaining the image features of the image to be recognized based on the feature extraction network, the recognition result of the image to be recognized can be determined according to the image features. In this process, the image features of the image to be detected can be directly input into the classification network to obtain the recognition result of the image features of the image to be detected. It is also possible to construct the image features of the template images for each target type label respectively, and compare the image features with the image features of the template images of each target type label to obtain the recognition result of the image features of the image to be detected. The embodiments of the present invention do not limit this.

[0057] The image recognition method provided by the embodiment of the present invention uses the first sample image carrying the target category label and the target segmentation label, and jointly trains the feature extraction network obtained by the initial semantic segmentation network to extract features from the image to be recognized, and obtains the recognition result according to the extracted image features, realizing that the feature extraction network can learn more feature information of the target area during the training process, reducing the mislearning of background information, improving the accuracy of the feature extraction network for extracting the features of the target area, and further improving the accuracy of the classification and recognition of the target area.

[0058] Based on the above embodiment, Figure 2 is a schematic flowchart of the feature extraction network training method provided by the present invention, as Figure 2 shown, the training steps of the feature extraction network include:

[0059] Step 210, based on the initial feature extraction network, determine the initial features of the first sample image;

[0060] Step 220, based on the similarity between the initial features of the first sample images under the same target category, and / or, the similarity between the initial features of the first sample images under different target categories, determine the first loss;

[0061] Step 230, based on the initial semantic segmentation network and the target segmentation label, determine the second loss;

[0062] Step 240, based on the first loss, perform parameter iteration on the initial feature extraction network, and based on the second loss, perform parameter iteration on the initial feature extraction network and the initial semantic segmentation network to obtain the feature extraction network.

[0063] Considering that the traditional feature extraction network requires a large amount of data for each training, and when a new target category is added, for example, when a new type of work clothes is added, the feature extraction network needs to be retrained, the training cost is very high, and the portability is also poor, and it cannot be quickly applied to work clothes types outside other scenarios and datasets. Therefore, the embodiment of the present invention constructs an initial feature extraction network based on metric learning.

[0064] Specifically, the initial feature extraction network is the initial network that realizes image feature extraction. Input the first sample image into the initial feature extraction network to obtain the initial features of the first sample image output by the initial feature extraction network. Under the idea of metric learning, the first loss can be determined according to the similarity between the initial features of the first sample images under the same target category, and / or, the similarity between the initial features of the first sample images under different target categories, and the parameters of the initial feature extraction network are iterated according to the first loss. The similarity between the features here can be calculated through a distance function or a cosine function, etc.

[0065] It should be noted that the greater the similarity between the initial features of the first sample images under the same target category, the better the initial feature extraction network's ability to extract the commonalities between the image features under the same target category, and thus the smaller the first loss calculated therefrom. The smaller the similarity between the initial features of the first sample images under the same target category, the worse the initial feature extraction network's ability to extract the commonalities between the image features under the same target category, and thus the larger the first loss calculated therefrom. The smaller the similarity between the initial features of the first sample images under different target categories, the better the initial feature extraction network's ability to extract the differences between the image features under the same target category, and thus the smaller the first loss calculated therefrom. The greater the similarity between the initial features of the first sample images under different target categories, the worse the initial feature extraction network's ability to extract the differences between the image features under the same target category, and thus the larger the first loss calculated therefrom.

[0066] That is, the obtained first loss here reflects the initial feature extraction network's ability to extract the feature commonalities of the first sample images under the same target category and / or the feature differences under different target categories.

[0067] In addition, the initial features of the first sample images are input into the initial semantic segmentation network to obtain the initial segmentation result output by the initial semantic segmentation network. The initial segmentation result here reflects the result of semantic segmentation of the first sample images based on the current network parameters. According to the initial segmentation result and the target segmentation label, the second loss can be determined; and the parameters of the initial feature extraction network and the initial semantic segmentation network are iterated according to the second loss.

[0068] It should be noted that the second loss, that is, the difference between the initial segmentation result and the target segmentation label, reflects whether the initial segmentation result obtained based on the initial feature extraction network and the initial semantic segmentation network conforms to the true target segmentation label. The smaller the second loss, the closer the initial segmentation result is to the target segmentation label.

[0069] Through the above first loss and second loss, the parameter iteration of the initial feature extraction network and the initial semantic segmentation network can be realized. Until both the first loss and the second loss converge, the feature extraction network can be obtained.

[0070] Here, the convergence of the first loss indicates that the initial feature extraction network has the ability to extract the common feature of images under the same target category and the difference feature of images under different target categories. The convergence of the second loss can indicate that the image features extracted by the initial feature extraction network are more inclined to the target area, that is, the image features highlight the features of the target area part more, and reduce the influence of the features of the non-target area part on the initial feature extraction network.

[0071] The image recognition method provided by the embodiments of the present invention is based on the idea of metric learning. The first loss is calculated using the similarity between features to achieve network training. When a new target category is added, only incremental training of the new target variety is required, enabling the feature extraction network to learn the mapping relationship from the images of the new target category to the vector space, reducing the training cost of the feature extraction network and improving the portability of the feature extraction network.

[0072] Based on the above embodiments, Figure 3 is a schematic diagram of the network structures of the feature extraction network and the semantic segmentation network provided by the present invention. As Figure 3 shown, during the training process of the feature extraction network, the output of the initial feature extraction network is connected to the input of the initial semantic segmentation network. The output of the initial feature extraction network, that is, the initial features, can calculate the first loss after passing through the fully connected layer, and the initial features can also be used as the input of the initial semantic segmentation network. The initial semantic segmentation network thus outputs the initial segmentation result and calculates the second loss therefrom.

[0073] Based on the above embodiments, Figure 4 is a schematic flowchart of the first loss determination method provided by the present invention. As Figure 4 shown, step 220 includes:

[0074] Step 221, determining multiple batches of sample images based on the first sample image;

[0075] Step 222, constructing the first positive and negative sample pairs and the second positive and negative sample pairs based on the initial features of the sample images in the current batch and the target category labels corresponding to the sample images in the current batch, as well as the initial features of the sample images in the batches before the current batch and the target category labels corresponding to the sample images in the batches before the current batch. Both sample images in the first positive and negative sample pairs come from the current batch, and one sample image in the second positive and negative sample pairs comes from the current batch and the other sample image comes from the batches before the current batch;

[0076] Step 223, determining the first loss based on the similarities of the first positive and negative sample pairs and the second positive and negative sample pairs.

[0077] Considering that the key to deep metric learning is the mining of hard example samples, and when mining hard example samples in each mini-batch of samples, whether by constructing positive and negative sample pairs within the mini-batch, measuring their difficulty from various angles, and then giving higher weights to the hard sample pairs, or considering the difficulty when generating the mini-batch, it will be affected by the size of the mini-batch. And the size of the mini-batch is limited by the hardware device and cannot be infinitely expanded. Even if the size of the mini-batch can be expanded by adding hardware devices, the training efficiency will be affected by the communication between hardware devices. Therefore, the embodiment of the present invention introduces cross mini-batch, that is, constructs a mini-batch queue of historical batches, and jointly forms positive and negative sample pairs with the sample images in the current mini-batch.

[0078] Specifically, first divide the first sample image into multiple batches of sample images according to the size of the mini-batch, and input them into the initial feature extraction network for training in batches. After the sample images of the current batch are input into the initial feature extraction network, the first positive and negative sample pairs and the second positive and negative sample pairs are constructed with the sample images of the batches before the current batch, and then the first loss is calculated through the first positive and negative sample pairs and the second positive and negative sample pairs. Among them, the first positive and negative sample pairs both come from the sample images of the current batch, and for the second positive and negative sample pairs, one image comes from the sample images of the current batch and the other image comes from the images of the batches before the current batch.

[0079] It should be noted that the batches before the current batch refer to a preset fixed number of batches before the current batch. Considering that training the initial feature extraction network with diverse positive and negative sample pairs can obtain better training effects, the sample images of the batches before the current batch are stored in the form of a queue. When the number of elements in the queue reaches the preset fixed number of batches, after the sample images of the current batch are inserted into the tail of the queue, the sample images of the head batch (the batch with the earliest enqueue time) will be deleted. For example, if the preset fixed number of batches is 10 and the current batch is the 15th batch, then the sample images in the batches before the current batch are the sample images of the 5th batch to the 14th batch. After the sample images of the current batch are inserted into the tail of the queue, the sample images of the 5th batch will be deleted from the queue, that is, now the queue stores the sample images of the 6th batch to the 15th batch.

[0080] In the image recognition method provided by the embodiments of the present invention, during the training process of the initial feature extraction network, positive and negative sample pairs are constructed through the sample images of the current batch and the sample images of the batches before the current batch. It is realized that during the training process, the initial feature extraction network learns more positive and negative sample pairs and hard example sample pairs, thereby improving the training effect of the initial feature extraction network, and further improving the accuracy of the feature extraction network in recognizing target categories.

[0081] Based on the above embodiments, Figure 5 is a schematic flowchart of the method for adaptively constructing the second positive and negative sample pairs provided by the embodiments of the present invention. As Figure 5 shown, the step of constructing the second positive and negative sample pairs in step 322 includes:

[0082] Step 510, determining a sampling ratio based on the similarity between the initial features of the sample images of the current batch and the similarity between the initial features of the sample images of the same batch in the batches before the current batch;

[0083] Step 520, constructing the second positive and negative sample pairs based on the initial features of the sample images of the current batch and the initial features of the sample images of the batches before the current batch, and the sampling ratio.

[0084] Considering that in the initial stage of the training of the initial feature extraction network, the situation of feature deviation of the samples is very serious, that is, the parameters of the initial feature extraction network are not stable enough and are far from the target network. In the initial stage of training, the initial features of the sample images output by the initial feature extraction network are relatively unreliable. As the parameters of the initial feature extraction network are continuously iterated during the training process, the reliability of the initial features will gradually increase, that is, the reliability of the initial features of the sample images stored in the sample image queue of the batches before the current batch will gradually increase. Therefore, the embodiments of the present invention introduce an adaptive weighting method to construct the second positive and negative sample pairs.

[0085] Specifically, calculate the similarity of the initial features of the sample images in the current batch to obtain the similarity between the initial features of the sample images of the same target category in the current batch, and calculate the similarity calculation value of the sample images of the same target category in the current batch, denoted as F1, based on the similarity between all the initial features of the sample images of the same target category obtained in the current batch; and the similarity of the initial features of the sample images of different target categories, and calculate the similarity calculation value of the sample images of different target categories in the current batch, denoted as F2, based on the similarity between all the initial features of the sample images of different target categories obtained in the current batch. At the same time, calculate the similarity of the initial features of the sample images in the batch before the current batch to obtain the similarity of the initial features of the sample images of the same target category in the batch before the current batch, and calculate the similarity calculation value of the sample images of the same target category in the batch before the current batch, denoted as H1, based on the similarity between all the initial features of the sample images of the same target category obtained in the batch before the current batch; and the similarity of the initial features of the sample images of different target categories, and calculate the similarity calculation value of the sample images of different target categories in the batch before the current batch, denoted as H2, based on the similarity between all the initial features of the sample images of different target categories obtained in the batch before the current batch. Use numerical statistical methods to calculate the sampling ratio for F1, F2, H1, and H2, that is, the ratio of the sampling quantity from the current batch to the sampling quantity from the batch before the current batch. Finally, sample from the initial features of the sample images in the current batch and the initial features of the sample images in the batch before the current batch according to the sampling ratio, and construct the second positive and negative sample pairs based on the sampling results. Here, the sampling ratio also represents the ratio of the second positive and negative sample pairs.

[0086] It should be noted that the similarity calculation value can be the mean, standard deviation, variance, etc. of the similarities of the initial features. The numerical statistical method can first calculate the ratio of the similarity calculation value H1 of the current batch under the same target category to the similarity calculation value F1 of the batch before the current batch, denoted as T1, and calculate the ratio of the similarity calculation value H2 of the current batch under different target categories to the similarity calculation value F2 of the batch before the current batch, denoted as T2, and then calculate the ratio of T1 to T2 to obtain the sampling ratio. The embodiments of the present invention do not limit this.

[0087] Construct a queue in the feature extraction network to store the initial features of the sample images in the batch before the current batch. In each subsequent iteration process, add the initial features of the sample images in the mini-Batch of the current batch to the queue, and delete the initial features of the sample images in the earliest mini-Batch from the queue to ensure that a certain number of initial features of the sample images available for comparison are stored in the queue and these initial features are all the latest.

[0088] Figure 6It is a schematic flowchart of the feature extraction network learning process provided by the present invention. As Figure 6 shown, the slanted rectangle represents the initial features of the sample images in the current batch, the blank rectangle represents the initial features of the sample images in the batches before the current batch, and Q represents the initial feature image queue of the superimposed sample images in the current batch and the batches before the current batch.

[0089] The image recognition method provided by the embodiments of the present invention calculates the similarities of the same target category and different target categories respectively through the initial features of the sample images in the current batch and the initial features of the sample images in the batches before the current batch, realizes the adaptive adjustment of the sampling ratios of the current batch and the batches before the current batch, reduces the influence of the feature shift generated during training on the training results, realizes the construction of a second pair of positive and negative samples with higher stability, further improves the training effect of the initial feature extraction network, and further improves the accuracy of the target category recognition by the feature extraction network.

[0090] Based on the above embodiments, step 110 includes:

[0091] Obtain the original image, perform target region detection on the original image, and use the target region in the original image as the image to be recognized.

[0092] It should be noted that the original image may contain one or more target regions. Each target region detected in the original image is intercepted with a rectangular frame of the same size to obtain each target region sub-image, and each target region sub-image is used as the image to be recognized.

[0093] Based on the above embodiments, step 130 includes:

[0094] Determine the recognition result of the image to be recognized based on the similarity between the image features and the sample image features of the second sample image corresponding to the target category.

[0095] Considering that the deep metric learning method can map the original features of an image to a vector feature in the vector space, that is, the image features, and can determine the similarity between two images based on the distance between the image features of the two images. Therefore, the embodiments of the present invention introduce the second sample image corresponding to each target category as a template, and thus obtain the recognition result of the image features of the image to be recognized, which can avoid the training work of the classification network and can efficiently obtain the recognition result of the image features.

[0096] Specifically, after obtaining the feature extraction network, the second sample images corresponding to each target category are respectively input into the feature extraction network, and the sample image features of the second sample images are obtained. Then, taking the sample image features of the second sample images as templates, the image features obtained by inputting the image to be recognized into the feature extraction network are respectively calculated for similarity with the sample image features of the second sample images corresponding to each target category, so as to obtain the similarity between the image features and the sample image features corresponding to each target category. And according to a preset similarity threshold, the recognition result of the image to be recognized is obtained.

[0097] It should be noted that to obtain the recognition result of the image to be recognized according to the preset similarity threshold, the similarities between the image features of the image to be recognized and the sample image features corresponding to each target category can be sorted from high to low first, and the highest similarity is compared with the preset similarity threshold. If it exceeds the preset similarity threshold, the target category corresponding to the highest similarity is taken as the recognition result of the image to be recognized. If it does not exceed, it means the recognition fails, that is, there is no target under any target category in the image to be recognized. For example, in the work uniform detection scenario, the recognition fails either because the worker does not wear a work uniform or because the worker wears a new type of work uniform, and the feature extraction network needs to perform incremental training on the new type of work uniform.

[0098] In addition, the second sample images can be from the first sample images or can be obtained by re - shooting, and the embodiments of the present invention do not limit this. Among them, the second sample images corresponding to each target category include images with one shooting angle or multiple shooting angles, and the embodiments of the present invention do not limit this.

[0099] Based on the above - mentioned embodiments, the second sample images include images taken from different shooting angles under the same target category. In particular, in the work uniform detection scenario, different shooting angles can include: the front of the work uniform, the side of the work uniform, and the back of the work uniform. The method provided by the embodiments of the present invention, through the setting of the second sample images with multiple shooting angles, can ensure that the images to be recognized taken at each angle can be successfully recognized when recognizing based on image features.

[0100] Figure 7 is a schematic flowchart of the feature extraction network test method provided by the present invention. As Figure 7 shown, the method includes:

[0101] Step 710, select the registration images of work uniforms for each target category.

[0102] Select about three second sample images for each target category, and use the second sample images as registered images. The registered images are required to have relatively complete work uniform features, covering the front, side, and back of the work uniform, which can fully display the features of the work uniform in all aspects. Each set of work uniform images Φ can be expressed as where R represents the registered image, i represents the registered image category, the superscript number 1 represents the front of the work uniform, 2 represents the side of the work uniform, and 3 represents the back of the work uniform.

[0103] Step 720: Output the work uniform feature value.

[0104] Input the registered images of each target category into the feature extraction network, output the image features of the registered images, and store the image features of each registered image:

[0105]

[0106] where F represents the image features of each work uniform image, and f represents the network convolution operation.

[0107] In particular, the image features of the registered images can be 128-dimensional vector features.

[0108] Step 730: Use a similarity function to calculate the similarity between the image features of the image to be recognized and the registered images.

[0109] Input the image to be recognized into the feature extraction network to obtain the image features T output by the feature extraction network, and use a similarity function to calculate the similarity between the image features of the image to be recognized and the registered images.

[0110] In particular, the similarity function can be a cosine similarity function, and the calculation formula is:

[0111]

[0112] where S represents the similarity between the test image and each registered image.

[0113] Step 740: By comparing the similarities in the S set with a preset similarity threshold, the target category of the image to be recognized, that is, the work uniform category, can be obtained.

[0114] Next, the image recognition device provided by the present invention will be described. The image recognition device described below can be correspondingly referred to the image recognition method described above.

[0115] Figure 8 is a schematic structural diagram of the image recognition device provided by the present invention. As Figure 8 shown, the device includes: a determination module 810, an extraction module 820, and a recognition module 830.

[0116] Among them,

[0117] Determination module 810, configured to determine an image to be recognized;

[0118] Extraction module 820, configured to extract features of the image to be recognized based on a feature extraction network, to obtain image features of the image to be recognized, where the feature extraction network is trained jointly with an initial semantic segmentation network based on a first sample image carrying a target category label and a target segmentation label, and the initial semantic segmentation network performs semantic segmentation based on features extracted by the feature extraction network;

[0119] Recognition module 830, configured to determine a recognition result of the image to be recognized based on the image features.

[0120] In an embodiment of the present invention, through the determination module, configured to determine an image to be recognized; the extraction module, configured to extract features of the image to be recognized based on a feature extraction network, to obtain image features of the image to be recognized, where the feature extraction network is trained jointly with an initial semantic segmentation network based on a first sample image carrying a target category label and a target segmentation label, and the initial semantic segmentation network performs semantic segmentation based on features extracted by the feature extraction network; the recognition module, configured to determine a recognition result of the image to be recognized based on the image features, it is realized that during the training process of the feature extraction network, more feature information of the target region can be learned, the mislearning of background information is reduced, the accuracy of the feature extraction network for extracting features of the target region is improved, and further the accuracy of classifying and recognizing the target region is improved.

[0121] Based on any of the above embodiments, the extraction module 820 includes a training module, and the module includes:

[0122] Initial feature extraction sub-module, configured to determine initial features of the first sample image based on an initial feature extraction network;

[0123] First loss calculation sub-module, configured to determine a first loss based on the similarity between the initial features of the first sample images under the same target category, and / or, the similarity between the initial features of the first sample images under different target categories;

[0124] Second loss calculation sub-module, configured to determine a second loss based on the initial semantic segmentation network and the target segmentation label;

[0125] Parameter iteration sub-module, configured to perform parameter iteration on the initial feature extraction network based on the first loss, and perform parameter iteration on the initial feature extraction network and the initial semantic segmentation network based on the second loss, to obtain a feature extraction network.

[0126] Based on any of the above embodiments, the first loss calculation sub-module includes:

[0127] A sample set splitting sub-module, configured to determine multiple batches of sample images based on a first sample image;

[0128] A sample pair constructing sub-module, configured to construct a first positive and negative sample pair and a second positive and negative sample pair based on the initial features of the sample images in the current batch and the corresponding target category labels of the sample images in the current batch, and the initial features of the sample images in the batches before the current batch and the corresponding target category labels of the sample images in the batches before the current batch. Both images in the first positive and negative sample pair come from the current batch, and one sample image in the second positive and negative sample pair comes from the current batch, and the other sample image comes from the batches before the current batch;

[0129] A first loss determining sub-module, configured to determine a first loss based on the similarities of the first positive and negative sample pair and the second positive and negative sample pair.

[0130] Based on any of the above embodiments, the sample pair constructing sub-module includes:

[0131] A sampling ratio determining sub-module, configured to determine a sampling ratio based on the similarities between the initial features of the sample images in the current batch and the similarities between the initial features of the sample images in the same batch in the batches before the current batch;

[0132] A second positive and negative sample pair constructing sub-module, configured to construct a second positive and negative sample pair based on the initial features of the sample images in the current batch and the initial features of the sample images in the batches before the current batch, and the sampling ratio.

[0133] Based on any of the above embodiments, a determining module 810 is configured to obtain an original image, perform target region detection on the original image, and use the target region in the original image as an image to be recognized.

[0134] Based on any of the above embodiments, a recognition module 830 is configured to determine a recognition result of the image to be recognized based on the similarity between the image feature and the sample image feature of the second sample image corresponding to the target category.

[0135] Based on any of the above embodiments, the second sample images in the recognition module 830 include images taken from different shooting angles under the same target category.

[0136] Figure 9 Illustrated is a schematic physical structure diagram of an electronic device, such as Figure 9As shown in the figure, the electronic device may include: a processor 910, a communications interface 920, a memory 930, and a communication bus 940. Among them, the processor 910, the communications interface 920, and the memory 930 complete communication with each other through the communication bus 940. The processor 910 may call the logical instructions in the memory 930 to execute an image recognition method, which includes: determining an image to be recognized; based on a feature extraction network, extracting features from the image to be recognized to obtain the image features of the image to be recognized. The feature extraction network is trained jointly with an initial semantic segmentation network based on the first sample image carrying the target category label and the target segmentation label, and the initial semantic segmentation network performs semantic segmentation based on the features extracted by the feature extraction network; based on the image features, determining the recognition result of the image to be recognized.

[0137] In addition, when the logical instructions in the above-mentioned memory 930 can be implemented in the form of software functional units and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0138] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image recognition method provided by the above-mentioned various methods. The method includes: determining an image to be recognized; based on a feature extraction network, extracting features from the image to be recognized to obtain the image features of the image to be recognized. The feature extraction network is trained jointly with an initial semantic segmentation network based on the first sample image carrying the target category label and the target segmentation label, and the initial semantic segmentation network performs semantic segmentation based on the features extracted by the feature extraction network; based on the image features, determining the recognition result of the image to be recognized.

[0139] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the image recognition method provided by the above-mentioned various methods. The method includes: determining an image to be recognized; based on a feature extraction network, extracting features from the image to be recognized to obtain the image features of the image to be recognized. The feature extraction network is trained jointly with an initial semantic segmentation network based on the first sample images carrying target category labels and target segmentation labels, and the initial semantic segmentation network performs semantic segmentation based on the features extracted by the feature extraction network; based on the image features, determining the recognition result of the image to be recognized.

[0140] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0141] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solutions, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or equivalently replace some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An image recognition method, characterized in that, Including: Determine the image to be recognized; Based on the feature extraction network, extract features from the image to be recognized to obtain the image features of the image to be recognized. The feature extraction network is trained jointly with an initial semantic segmentation network based on the first sample images carrying target class labels and target segmentation labels, and the initial semantic segmentation network performs semantic segmentation based on the features extracted by the feature extraction network; Based on the image features, determine the recognition result of the image to be recognized; The training steps of the feature extraction network include: Based on the initial feature extraction network, determine the initial features of the first sample images; Based on the similarity between the initial features of the first sample images under the same target class, and / or the similarity between the initial features of the first sample images under different target classes, determine the first loss; Based on the initial semantic segmentation network and the target segmentation labels, determine the second loss; Based on the first loss, perform parameter iteration on the initial feature extraction network, and based on the second loss, perform parameter iteration on the initial feature extraction network and the initial semantic segmentation network to obtain the feature extraction network.

2. The image recognition method according to claim 1, characterized in that, The determining the first loss based on the similarity between the initial features of the first sample images under the same target class, and / or the similarity between the initial features of the first sample images under different target classes, includes: Based on the first sample images, determine multiple batches of sample images; Based on the initial features of the sample images in the current batch and the corresponding target class labels of the sample images in the current batch, and the initial features of the sample images in the batches before the current batch and the corresponding target class labels of the sample images in the batches before the current batch, construct the first positive and negative sample pairs and the second positive and negative sample pairs. The two images in the first positive and negative sample pairs are both from the current batch, and one sample image in the second positive and negative sample pairs is from the current batch and the other sample image is from the batches before the current batch; Based on the similarity of the first positive and negative sample pairs and the second positive and negative sample pairs, determine the first loss.

3. The image recognition method according to claim 2, wherein The step of constructing the second positive and negative sample pairs includes: Based on the similarity between the initial features of the sample images in the current batch, and the similarity between the initial features of the sample images in the same batch in the batches before the current batch, determine the sampling ratio; Based on the initial features of the sample images in the current batch and the initial features of the sample images in the batches before the current batch, and the sampling ratio, construct the second positive and negative sample pairs.

4. The image recognition method according to any one of claims 1 to 3, characterized in that The determining the image to be recognized includes: Obtain the original image, perform target region detection on the original image, and use the target region in the original image as the image to be recognized.

5. The image recognition method according to any one of claims 1 to 3, characterized in that, The determining the recognition result of the image to be recognized based on the image features includes: Based on the similarity between the image features and the sample image features of the second sample images corresponding to the target classes, determine the recognition result of the image to be recognized.

6. The image recognition method according to claim 5, wherein The second sample images include images taken at different shooting angles under the same target class.

7. An image recognition device, characterized in that, Including: A determination module, configured to determine an image to be recognized; An extraction module, configured to perform feature extraction on the image to be recognized based on a feature extraction network, so as to obtain image features of the image to be recognized, where the feature extraction network is trained jointly with an initial semantic segmentation network based on a first sample image carrying a target category label and a target segmentation label, and the initial semantic segmentation network performs semantic segmentation based on features extracted by the feature extraction network; A recognition module, configured to determine a recognition result of the image to be recognized based on the image features; The training steps of the feature extraction network include: Based on an initial feature extraction network, determining initial features of the first sample image; Determining a first loss based on the similarity between the initial features of the first sample images under the same target category and / or the similarity between the initial features of the first sample images under different target categories; Determining a second loss based on the initial semantic segmentation network and the target segmentation label; Performing parameter iteration on the initial feature extraction network based on the first loss, and performing parameter iteration on the initial feature extraction network and the initial semantic segmentation network based on the second loss to obtain the feature extraction network.

8. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the image recognition method according to any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the image recognition method according to any one of claims 1 to 6.

Images