Image recognition model training method, device, electronic device and storage medium
By using the mutual supervision training method of twin networks in image recognition technology, the problem of insufficient robustness and accuracy of image recognition in 3D visual prevention and control scenarios is solved, and the accurate recognition of object type, size and three-dimensional information is achieved.
Patent Information
- Application Number
- CN202110720618.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-06-28
- Publication Date
- 2024-12-03
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing image recognition technology has shortcomings in terms of robustness and accuracy, especially in 3D visual prevention and control scenarios, it is difficult to effectively identify the type, size and three-dimensional information of objects.
By obtaining the training data set, including the original image and the transformed image, input the encoder and decoder of the twin network respectively, performing mutual supervision training, and correcting the initial twin network to obtain a more accurate and robust image recognition network.
It improves the accuracy and robustness of image recognition, and can accurately identify the type, size and three-dimensional information of objects in 3D visual prevention and control scenarios, and is suitable for fields such as intelligent transportation.
Smart Images

Figure CN113591566B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence, specifically computer vision and deep learning technology, and can be used specifically in 3D vision prevention and control scenarios. Background Art
[0002] Image recognition refers to the technology of using computers to process, analyze and understand images to identify targets and objects of various different patterns. It is a practical application of deep learning algorithms. With the development of science and technology, image recognition is increasingly used in people's lives.
[0003] Therefore, how to improve the robustness of image recognition is an urgent problem to be solved. Summary of the invention
[0004] The present application provides a method, device, electronic device and storage medium for training an image recognition model.
[0005] According to one aspect of the present application, a method for training an image recognition model is provided, comprising:
[0006] Acquire a training data set, wherein the training data set includes a first image set, a second image set, and a labeling result corresponding to each first image in the first image set, wherein each second image in the second image set is generated after the corresponding first image is subjected to a first transformation;
[0007] Input any first image and the corresponding second image into the first encoder and the second encoder in the initial twin network respectively to obtain a first feature map corresponding to the any first image and a second feature map corresponding to the corresponding second image;
[0008] Inputting the first feature map and the second feature map into the first decoder and the second decoder in the initial twin network respectively to obtain a first prediction result and a second prediction result;
[0009] According to the second feature map, the first feature map, the first prediction result, the second prediction result and the annotation result corresponding to any one of the first images, the initial twin network is corrected to obtain a first image recognition network and a second image recognition network.
[0010] According to another aspect of the present application, a training device for an image recognition model is provided, comprising:
[0011] A first acquisition module is used to acquire a training data set, wherein the training data set includes a first image set, a second image set, and a labeling result corresponding to each first image in the first image set, wherein each second image in the second image set is generated after the corresponding first image is subjected to a first transformation;
[0012] A second acquisition module is used to input any first image and the corresponding second image into the first encoder and the second encoder in the initial twin network respectively, so as to obtain a first feature map corresponding to the any first image and a second feature map corresponding to the corresponding second image;
[0013] A third acquisition module, used to input the first feature map and the second feature map into the first decoder and the second decoder in the initial twin network respectively, so as to obtain a first prediction result and a second prediction result;
[0014] A training module is used to correct the initial twin network according to the second feature map, the first feature map, the first prediction result, the second prediction result and the annotation result corresponding to any one of the first images to obtain a first image recognition network and a second image recognition network.
[0015] According to another aspect of the present application, there is provided an electronic device, including:
[0016] at least one processor; and
[0017] a memory communicatively connected to the at least one processor; wherein,
[0018] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor so that the at least one processor can perform the method described in the above embodiment.
[0019] According to another aspect of the present application, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method according to the above embodiment.
[0020] According to another aspect of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method according to the above embodiment is implemented.
[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present application.
[0023] Figure 1 A flowchart of a method for training an image recognition model provided in an embodiment of the present application;
[0024] Figure 2 A flowchart of another method for training an image recognition model provided in an embodiment of the present application;
[0025] Figure 3 A flowchart of another method for training an image recognition model provided in an embodiment of the present application;
[0026] Figure 4 A flowchart of another method for training an image recognition model provided in an embodiment of the present application;
[0027] Figure 5 A training diagram of an image recognition model provided in an embodiment of the present application;
[0028] Figure 6 A schematic diagram of the structure of a training device for an image recognition model provided in an embodiment of the present application;
[0029] Figure 7 It is a block diagram of an electronic device used to implement the training method of the image recognition model of the embodiment of the present application. DETAILED DESCRIPTION
[0030] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
[0031] The following describes the image recognition model training method, device, electronic device and storage medium of the embodiments of the present application with reference to the accompanying drawings.
[0032] Artificial intelligence is a discipline that studies the use of computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level technical fields and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies include computer vision technology, speech recognition technology, natural language processing technology, as well as deep learning, big data processing technology, knowledge graph technology, and other major directions.
[0033] Computer vision is a science that studies how to make machines "see". It refers to using cameras and computers to replace human eyes to identify, track and measure targets, and further perform graphic processing so that the computer processes the images into images that are more suitable for human eye observation or transmission to instruments for detection.
[0034] Deep learning is a new research direction in the field of machine learning. Deep learning is the study of the inherent laws and representation levels of sample data. The information obtained in the learning process is of great help in the interpretation of data such as text, images and sounds. Its ultimate goal is to enable machines to have analytical learning capabilities like humans and to recognize data such as text, images and sounds.
[0035] Figure 1 A flowchart of a method for training an image recognition model provided in an embodiment of the present application.
[0036] The training method of the image recognition model of the embodiment of the present application can be executed by the training device of the image recognition model of the embodiment of the present application. The device can be configured in an electronic device to input the image and the corresponding transformed image into two encoders of the twin network respectively, and obtain two recognition networks through mutual supervision training, thereby improving the accuracy and robustness of recognition.
[0037] like Figure 1 As shown, the training method of the image recognition model includes:
[0038] Step 101: Acquire a training data set, wherein the training data set includes a first image set, a second image set, and a labeling result corresponding to each first image in the first image set.
[0039] In the present application, the first image set and the annotation results corresponding to each first image in the first image set can be obtained. Each first image can be obtained by photographing an object. For example, the first image set may include an image of an apple, an image of a chair, an image of a vehicle, etc. The annotation result corresponding to the first image here can be at least one of the following: the type of object, the size of the object, the three-dimensional center point of the object, the orientation angle of the object, the size of the two-dimensional detection frame of the object, etc. The size can refer to the physical size of the object, such as the length, width, height, etc. of the object.
[0040] In the present application, a recognition network for recognizing different information of an object can be trained based on different annotation results corresponding to the first image.
[0041] After acquiring the first image set, a first transformation may be performed on each first image in the first image set to generate a second image corresponding to each first image, and all second images constitute a second image set. That is, each second image in the second image set is obtained by performing the first transformation on the corresponding first image. The first transformation performed on the first image includes but is not limited to affine transformation, contrast transformation, color gamut space adjustment, etc.
[0042] In the present application, a training set can be obtained according to the first image set, the second image set, the annotation results corresponding to each first image in the first image set, and the like.
[0043] Step 102, input any first image and the corresponding second image into the first encoder and the second encoder in the initial twin network respectively to obtain a first feature map corresponding to any first image and a second feature map corresponding to the corresponding second image.
[0044] In the present application, the initial network used for training is an initial twin network, which may include two initial recognition networks. The two initial recognition networks may share weights, one initial recognition network includes a first encoder and a first decoder, and the other initial recognition network includes a second encoder and a second decoder.
[0045] After obtaining the training data set, any first image in the training data set can be input into the first encoder in the initial twin network, and the first encoder can perform feature extraction to obtain a first feature map corresponding to any first image output by the first encoder, and the second image corresponding to the first image can be input into the second encoder of the initial twin network, and the second encoder can perform feature extraction to obtain a second feature map corresponding to the second image output by the second encoder.
[0046] For example, after an image A is affine transformed, image A' is obtained. Image A can be input into the first encoder of the initial twin network to obtain the first feature map of image A, and image A' can be input into the second encoder of the initial twin network to obtain the second feature map of image A'.
[0047] Step 103, input the first feature map and the second feature map into the first decoder and the second decoder in the initial twin network respectively to obtain the first prediction result and the second prediction result.
[0048] When obtaining the first feature map and the second feature map, the first feature map can be input into the first decoder for decoding to obtain a first prediction result corresponding to the first image, and the second feature map can be input into the second decoder for decoding to obtain a second prediction result corresponding to the second image.
[0049] Among them, the first prediction result may include the size of the two-dimensional detection box of the object in the first image, the type of the object, the size of the object, the three-dimensional center point of the object, etc., and the second prediction result has the same type and quantity of prediction content as the first prediction result.
[0050] Step 104, according to the second feature map, the first feature map, the first prediction result, the second prediction result and the annotation result corresponding to any first image, the initial twin network is corrected to obtain the first image recognition network and the second image recognition network.
[0051] Since the first feature map and the second feature map are respectively input into the two encoders, and the second image is obtained by transforming the first image, the first feature map and the second feature map are also associated with each other. Therefore, the initial twin network can be corrected according to the first feature map, the second feature map, the first prediction result, and the second prediction result, that is, the two initial recognition networks of the initial twin network are corrected to obtain the first image recognition network and the second image recognition network.
[0052] For example, a loss function can be obtained based on the first feature map and the second feature map, a corresponding loss function can be obtained based on the first prediction result and the labeling result corresponding to the first image, and a loss function can be obtained based on the second prediction result and the labeling result corresponding to the first image. According to the three loss functions, the parameters of the initial twin network are adjusted, and the twin network after the parameter adjustment is continued to be adjusted until the twin network converges to obtain two image recognition networks.
[0053] In the present application, when training the initial twin network, deep learning can be used for training to obtain a first image recognition network and a second image recognition network.
[0054] After the training is completed, the image to be recognized can be input into the first image recognition network or the second image recognition network to obtain the recognition result of the image to be recognized. In other words, any image recognition network in the twin network can be used for image recognition.
[0055] The training method of the image recognition model of the embodiment of the present application can be based on the different annotation results of the first image to train a recognition network for identifying different information of the object. For example, if the annotation results corresponding to the first image include the type of the object and the size of the two-dimensional detection frame of the object, a recognition network for two-dimensional object detection can be trained using the training data set; if the annotation results corresponding to the first image include the type of the object, the three-dimensional size, the size of the two-dimensional detection frame of the object, the three-dimensional center point of the object, and the orientation angle of the object, then a recognition network for three-dimensional object detection can be trained using the training data set.
[0056] The training method of the image recognition model of the present application can be applied to 3D visual prevention and control scenarios. By collecting images within the prevention and control area, the trained image recognition network can be used to identify the type, size, three-dimensional center point and other information of each object in the image. Based on the information of each object obtained by identification, corresponding prevention and control measures can be collected.
[0057] The training method of the image recognition model of the embodiment of the present application can also be applied in the field of intelligent transportation. For example, images of the surrounding area of a vehicle can be collected, and an image recognition network can be used to identify the type, size, orientation angle and other information of each object in the collected image, and use this information to assist vehicle driving.
[0058] In an embodiment of the present application, the first image and the second image obtained after the first image is transformed are respectively input into the first encoder and the second encoder of the initial twin network to obtain the first feature map and the second feature map, and the first feature map and the second feature map are respectively input into the first decoder and the second decoder in the initial twin network to obtain the first prediction result and the second prediction result. Based on the second feature map, the first feature map, the first prediction result, the second prediction result and the annotation result corresponding to any first image, the first image recognition network and the second image recognition network are trained. Thus, by inputting the image and the corresponding transformed image into the two encoders of the twin network respectively, two recognition networks are obtained through mutual supervision training, which improves the accuracy and robustness of recognition.
[0059] Figure 2 A flowchart of another method for training an image recognition model provided in an embodiment of the present application.
[0060] like Figure 2 As shown, the training method of the image recognition model includes:
[0061] Step 201 : obtaining a training data set, wherein the training data set includes a first image set, a second image set, and a labeling result corresponding to each first image in the first image set.
[0062] Step 202, input any first image and the corresponding second image into the first encoder and the second encoder in the initial twin network respectively to obtain a first feature map corresponding to any first image and a second feature map corresponding to the corresponding second image.
[0063] Step 203, input the first feature map and the second feature map into the first decoder and the second decoder in the initial twin network respectively to obtain the first prediction result and the second prediction result.
[0064] In the present application, steps 201 to 203 are similar to the above-mentioned steps 101 to 103, and thus will not be described in detail here.
[0065] Step 204: Perform a second transformation on the second feature map to obtain a third feature map, wherein the second transformation and the first transformation are inverse transformations of each other.
[0066] Since the second image is generated by the corresponding first image after the first transformation, in the present application, the second feature can be subjected to a second transformation to obtain a third feature map, wherein the second transformation and the first transformation are inverse transformations of each other, that is, the second transformation is the inverse transformation of the first transformation.
[0067] Step 205: Determine a first difference between the third feature map and the first feature map.
[0068] Since the second image is generated by the corresponding first image after the first transformation, and the third feature map is obtained by the second feature map after the inverse transformation of the first transformation, there is a certain similarity between the first feature map and the third feature map.
[0069] In the present application, the first difference between the third feature map and the first feature map can be determined, and the first difference can be used as a basis for correcting the initial twin network. For example, the absolute value of the difference between the third feature map and the first feature map is obtained and used as a loss function.
[0070] Step 206: Determine a second difference between the first prediction result and the annotation result corresponding to any first image.
[0071] In the present application, if the annotation results include annotation results of multiple information in the first image, then the difference between the prediction result and the annotation result corresponding to each information can be determined, and the sum of the differences of all information can be calculated as the second difference between the first prediction result corresponding to any first image and the corresponding annotation result.
[0072] For example, the annotation result includes the size of the object's two-dimensional detection box, the size of the object, the object's orientation angle and the type of the object. Then, the difference between the predicted size of the two-dimensional detection box and the size of the object's two-dimensional detection box in the annotation result, the difference between the predicted size of the object and the size of the annotation result, the difference between the predicted orientation angle of the object and the orientation angle of the object in the annotation result, and the difference between the predicted type of the object and the type of the object in the annotation result can be fused, and the fusion result can be used as the second difference between the first prediction result and the annotation result.
[0073] Step 207: Determine a third difference between the second prediction result and the annotation result corresponding to any one of the first images.
[0074] In the present application, the method for determining the third difference is similar to the method for determining the second difference, so it will not be repeated here.
[0075] Step 208, based on the first difference, the second difference and the third difference, the initial twin network is modified to obtain a first image recognition network and a second image recognition network.
[0076] In the present application, the corresponding loss functions can be obtained according to the first difference, the second difference and the third difference respectively, and the sum of the three loss functions can be calculated, and the sum of the loss functions can be determined as the final loss value. When the loss value is greater than the threshold, the initial twin network is corrected based on the final loss value. Thus, by fusing the first difference, the second difference and the third difference, the initial twin network is corrected based on the final loss value determined by the fusion, so that the outputs of the two recognition networks are similar, thereby obtaining two image recognition networks.
[0077] Alternatively, the first difference, the second difference, and the third difference may correspond to different weights, and the weighted sum of the loss function corresponding to the first difference, the loss function corresponding to the second difference, and the loss function corresponding to the third difference may be calculated, and the final loss value may be determined based on the weighted sum. When the loss value is greater than a threshold, the initial twin network may be corrected based on the final loss value. In this way, the personalized needs of training may be met.
[0078] In an embodiment of the present application, when the initial twin network is corrected to obtain the first image recognition network and the second image recognition network, the third feature map can be obtained by performing an inverse transformation of the first transformation on the second feature map. Based on the first difference between the third feature map and the first feature map, the second difference between the first prediction result and the annotation result, and the third difference between the second prediction result and the annotation result, the initial twin network is corrected, so that two image recognition networks can be obtained, and the output results of the two image recognition networks are similar, and then any image recognition network can be used for image recognition, thereby improving the accuracy and robustness of image recognition.
[0079] In order to further improve the accuracy of image recognition, in one embodiment of the present application, the coordinates of each pixel in the first image can be encoded to obtain a position vector, and the position vector is also input into the twin network to obtain a feature map and a prediction result. Figure 3 To explain, Figure 3 A flowchart of another method for training an image recognition model provided in an embodiment of the present application.
[0080] like Figure 3 As shown, the training method of the image recognition model includes:
[0081] Step 301 : obtaining a training data set, wherein the training data set includes a first image set, a second image set, and a labeling result corresponding to each first image in the first image set.
[0082] In the present application, step 301 is similar to the above step 101, so it will not be repeated here.
[0083] Step 302: Determine a first position vector corresponding to the first image according to the first coordinate value of each first pixel point in any first image.
[0084] In the present application, the first coordinate value of each first pixel in any first image can be encoded to obtain the position vector of each first pixel, thereby mapping the position information of the pixel to a point in a multidimensional space. When encoding the first coordinate value, sine coding, cosine coding, Taylor expansion method, learnable parameter method, etc. can be used.
[0085] In the present application, the position vectors of all first pixel points in the first image may be spliced to obtain the first position vector corresponding to the first image.
[0086] Step 303: Determine a second position vector corresponding to the second image according to the second coordinate value of each second pixel point in the corresponding second image.
[0087] In the present application, the method of determining the second position vector corresponding to the second image is similar to the method of determining the first position vector corresponding to the first image, so it will not be repeated here.
[0088] Step 304: input any first image and the first position vector into a first encoder to obtain a first feature map corresponding to any first image, and input the corresponding second image and the second position vector into a second encoder to obtain a second feature map corresponding to the second image.
[0089] In the present application, any first image and the first position vector corresponding to the first image can be input into the first encoder to obtain the first feature map corresponding to any first image, and the corresponding second image and the second position vector corresponding to the second image can be input into the second encoder to obtain the second feature map corresponding to the second image. Thus, high-dimensional feature information is added to the input of the encoder, which can improve the recognition accuracy of the twin network.
[0090] Step 305: Input the first feature map and the second feature map into the first decoder and the second decoder in the initial twin network respectively to obtain the first prediction result and the second prediction result.
[0091] Step 306, according to the second feature map, the first feature map, the first prediction result, the second prediction result and the annotation result corresponding to any first image, the initial twin network is corrected to obtain the first image recognition network and the second image recognition network.
[0092] In the present application, step 305-step 306 is similar to the above-mentioned step 103-step 104, so they will not be repeated here.
[0093] In an embodiment of the present application, when any first image and the corresponding second image are respectively input into the first encoder and the second encoder in the initial twin network, the first position vector corresponding to the first image is determined according to the first coordinate value of each first pixel point in any first image, and the second position vector corresponding to the second image is determined according to the second coordinate value of each second pixel point in the corresponding second image, and the first position vector and the first image are input into the first encoder together, and the second position vector and the second image are input into the second encoder together. Thus, by encoding the coordinate value of each pixel point in the first image and the corresponding second image, high-dimensional feature information is obtained, and high-dimensional feature information is added to the input of the first encoder and the second encoder, thereby improving the accuracy of image recognition.
[0094] In one embodiment of the present application, Figure 4 Determine the size of the object in the image in the manner shown. Figure 4 A flowchart of another method for training an image recognition model provided in an embodiment of the present application.
[0095] like Figure 4 As shown, the obtaining of the first prediction result and the second prediction result may include:
[0096] Step 401 , obtaining first probabilities that any first image belongs to each type of image, and first size deviation values corresponding to each type of image.
[0097] In the present application, first images of the same type in the training data set can be classified as the same category according to the type of the object in each first image, so that the image types included in the training data set can be obtained.
[0098] In the present application, the first decoder can decode the first feature map to obtain the first probability that any image belongs to each type of image, and the first size deviation value corresponding to each type of image.
[0099] For example, the training data set contains 10 categories of images. When obtaining the prediction results, you can obtain the probability that the image belongs to each of the 10 categories of images, and the corresponding size deviation value when it belongs to each category of images.
[0100] In the present application, if the size of the object in the annotation result is the length, width and height of the object, then the size deviation value in the prediction result includes the deviation value of the length, the deviation value of the width and the deviation value of the height.
[0101] Step 402, obtaining second probabilities that the corresponding second image belongs to each type of image, and second size deviation values corresponding to each type of image.
[0102] In the present application, the method for obtaining the second probability and the second size deviation value is similar to the above-mentioned method for obtaining the first probability and the first size deviation value, so it will not be repeated here.
[0103] Step 403: Determine the first target type to which any first image belongs according to each first probability.
[0104] In the present application, based on the first probabilities that any first image belongs to each type of image, the type corresponding to the first probability with the largest value can be used as the first target type to which any first image belongs.
[0105] For example, there are 10 types of first images in the training data set, and the probability that an image belongs to type a is the highest, so type a can be used as the first target type.
[0106] Step 404 , determining a first size of an object in any first image according to a first size deviation value corresponding to when any first image belongs to a first target type and a first reference size corresponding to the first target type.
[0107] In the present application, the reference size corresponding to each type can be determined in advance, so the first reference size corresponding to the first target type can be obtained, and the first size deviation value corresponding to the first image belonging to the first target type can be added to the first reference size to obtain the first size of the object in the first image.
[0108] In the present application, the annotation result may include the size and type of the object in each first image. When determining the reference size corresponding to each type of image, first images with the same type of object may be regarded as the same type of image according to the type of object in each first image, so that the sub-image set corresponding to each type of image may be determined. It is understood that the object type in each first image in each sub-image set is the same.
[0109] That is, the training data set may be divided into a plurality of sub-image sets according to the type of the object in each first image, and the object type of each first image in each sub-image set is the same.
[0110] After determining the sub-image set corresponding to each type of image, the sum of the sizes of objects in all first images in each sub-image set can be calculated, and the sum of the sizes can be divided by the number of first images in each sub-image set to obtain the reference size corresponding to each type of image.
[0111] In the present application, the training data set can be divided into multiple sub-image sets according to the size and type of the object in the first image in the annotation result, and the reference size corresponding to the object of the same type can be determined based on the size of the object in the first image of the same type, so that the size of the object can be determined based on the reference size corresponding to the type to which the first object belongs and the size deviation value corresponding to the type output by the decoder.
[0112] Step 405: Determine the second target type to which the corresponding second image belongs according to each second probability.
[0113] Step 406 , determining a second size of the object in the corresponding second image according to a second size deviation value corresponding to the second target type and a second reference size corresponding to the second target type when the corresponding second image belongs to the second target type.
[0114] In the present application, determining the second target type to which the second image belongs and the second size of the object in the second image is similar to determining the first target type and the first size as described above, and thus will not be described in detail herein.
[0115] After determining the first size of the object in the first image and the second size in the second image, the difference between the first size and the size of the object in the annotation result, and the difference between the second size and the size of the object in the annotation result can be determined. Thus, the initial twin network can be trained based on the size difference.
[0116] In an embodiment of the present application, when predicting the size of an object, the initial twin network can be used to obtain the first probability that the first image belongs to each type of image, and the first size deviation value corresponding to the first image when belonging to each type of image, and the second probability that the second image belongs to each type of image, and the second size deviation value corresponding to the second image when belonging to each type of image, and determine the first target type to which the first image belongs according to each first probability, determine the second target type to which the first image belongs according to each second probability, and determine the first size of the object according to the reference size and the first size deviation value corresponding to the first target type, and determine the second size of the object according to the reference size and the second size deviation value corresponding to the second target type. Thus, the size deviation value of the object can be predicted through the initial twin network, and the size of the object can be determined according to the size deviation value and the reference size corresponding to the predicted target type.
[0117] Combine the following Figure 5 Further explain the training method of the image recognition model of this application, Figure 5 A training diagram of an image recognition model provided in an embodiment of the present application.
[0118] Figure 5 In , the initial twin network has two branches, each of which includes an encoder and a decoder, where the first encoder shares weights with the second encoder. Figure 5 As shown, the coordinate value of each pixel in the image I can be encoded to determine the first position vector corresponding to I. The image I and the first position vector can be input into the first encoder of the initial twin network for feature extraction to obtain the first feature map fm1, and the image I can be transformed by f to obtain the image ,Right now Image and images The corresponding second feature vector is input into the second encoder for feature extraction to obtain the second feature map fm2. That is, high-dimensional feature information is added to the input of the first encoder and the second encoder.
[0119] Figure 5 In the example, the first feature map fm1 can be input into the first decoder to obtain the first prediction result, and the second feature map fm2 can be input into the second decoder to obtain the second prediction result. The first prediction result and the second prediction result may include the size of the two-dimensional detection frame of the object, the type of the object, the size of the object, the orientation angle of the object, and the three-dimensional center point of the object.
[0120] Afterwards, the loss function L can be determined based on the first prediction result and the annotation result corresponding to the image I. s1 , according to the second prediction result and the annotation result corresponding to image I, determine the loss function L s2 Among them, L s1 It can be the loss function L corresponding to the size of the object's two-dimensional detection box 2dbox , the loss function L corresponding to the type of object cls , the loss function L corresponding to the size of the object lwh , the loss function L corresponding to the orientation angle of the object angle , the loss function L corresponding to the three-dimensional center point of the object 3Dloc etc., namely L s1 =L 2dbox +L cls +L lwh +L angle +L 3Dloc In addition, the loss function L s2 With L s1 Similar, no further elaboration.
[0121] like Figure 5 As shown, the inverse transformation f of transformation f can also be applied to fm2 -1 , get the third feature map fm3, and calculate the loss function L based on fm1 and fm3 disturb =|fm3-fm1|, so that fm3 is as similar as possible to fm1.
[0122] Figure 5 In the example above, L s1 , L s2 and L disturb The sum of the two is used as the final loss function L all =L s1 +L s2 +L disturb , if L all is greater than the threshold, then according to L allThe initial twin network is modified to obtain the first image recognition network and the second image recognition network. After the training is completed, only a single branch network of the twin network can be retained, and a branch network in the twin network can be used to perform three-dimensional detection on the image to be recognized to obtain the detection result.
[0123] In order to implement the above embodiment, the embodiment of the present application also proposes a training device for an image recognition model. Figure 6 A schematic diagram of the structure of a training device for an image recognition model provided in an embodiment of the present application.
[0124] like Figure 6 As shown, the training device 600 of the image recognition model includes:
[0125] A first acquisition module 610 is used to acquire a training data set, wherein the training data set includes a first image set, a second image set, and a labeling result corresponding to each first image in the first image set, wherein each second image in the second image set is generated after the corresponding first image is subjected to a first transformation;
[0126] A second acquisition module 620 is used to input any first image and the corresponding second image into the first encoder and the second encoder in the initial twin network respectively, so as to obtain a first feature map corresponding to the any first image and a second feature map corresponding to the corresponding second image;
[0127] A third acquisition module 630, configured to input the first feature map and the second feature map into the first decoder and the second decoder in the initial twin network, respectively, to obtain a first prediction result and a second prediction result;
[0128] The training module 640 is used to modify the initial twin network according to the second feature map, the first feature map, the first prediction result, the second prediction result and the annotation result corresponding to any first image to obtain the first image recognition network and the second image recognition network.
[0129] In a possible implementation of the embodiment of the present application, the training module 640 includes:
[0130] a transformation unit, configured to perform a second transformation on the second feature map to obtain a third feature map, wherein the second transformation and the first transformation are inverse transformations of each other;
[0131] a first determining unit, configured to determine a first difference between the third characteristic map and the first characteristic map;
[0132] A second determining unit, configured to determine a second difference between the first prediction result and a labeling result corresponding to any one of the first images;
[0133] A third determining unit, configured to determine a third difference between the second prediction result and a labeling result corresponding to any one of the first images;
[0134] A training unit is used to modify the initial twin network based on the first difference, the second difference and the third difference to obtain a first image recognition network and a second image recognition network.
[0135] In a possible implementation of the embodiment of the present application, the training unit is used to:
[0136] fusing the first difference, the second difference, and the third difference to determine a loss value;
[0137] When the loss value is greater than a threshold, the initial twin network is modified based on the loss value to obtain a first image recognition network and a second image recognition network.
[0138] In a possible implementation of the embodiment of the present application, the second acquisition module 620 is used to:
[0139] Determine a first position vector corresponding to the first image according to a first coordinate value of each first pixel point in any one of the first images;
[0140] Determining a second position vector corresponding to the second image according to a second coordinate value of each second pixel point in the corresponding second image;
[0141] The any one first image and the first position vector are input into the first encoder, and the corresponding second image and the second position vector are input into the second encoder.
[0142] In a possible implementation of the embodiment of the present application, the third acquisition module 630 is used to:
[0143] Obtaining first probabilities that any of the first images belongs to each type of image, and first size deviation values corresponding to the first images when the first images belong to each type of image;
[0144] Obtaining second probabilities that the corresponding second images belong to each type of image, and second size deviation values corresponding to the second images when the corresponding second images belong to each type of image;
[0145] Determining a first target type to which any one of the first images belongs according to each of the first probabilities;
[0146] Determining a first size of an object in any of the first images according to a first size deviation value corresponding to when any of the first images belongs to the first target type and a first reference size corresponding to the first target type;
[0147] Determining, according to each of the second probabilities, a second target type to which the corresponding second image belongs;
[0148] The second size of the object in the corresponding second image is determined according to the second size deviation value corresponding to when the corresponding second image belongs to the second target type and the second reference size corresponding to the second target type.
[0149] In a possible implementation of the embodiment of the present application, the annotation result includes the size and type of the object in each first image, and the device may further include:
[0150] A first determination module is used to determine a sub-image set corresponding to each type of image according to the type of the object in each first image, wherein the type of the object in each first image in each sub-image set is the same;
[0151] The second determination module is used to determine the reference size corresponding to each type of image according to the size of the objects in each first image in each sub-image set.
[0152] In a possible implementation of the embodiment of the present application, the annotation result includes at least one of the following: the size of the two-dimensional detection box of the object, the size of the object, the three-dimensional center point of the object, the orientation angle of the object, and the type of the object.
[0153] It should be noted that the explanation of the aforementioned embodiment of the training method of the image recognition model is also applicable to the training device of the image recognition model of this embodiment, so it will not be repeated here.
[0154] In an embodiment of the present application, the first image and the second image obtained after the first image is transformed are respectively input into the first encoder and the second encoder of the initial twin network to obtain the first feature map and the second feature map, and the first feature map and the second feature map are respectively input into the first decoder and the second decoder in the initial twin network to obtain the first prediction result and the second prediction result. Based on the second feature map, the first feature map, the first prediction result, the second prediction result and the annotation result corresponding to any first image, the first image recognition network and the second image recognition network are trained. Thus, by inputting the image and the corresponding transformed image into the two encoders of the twin network respectively, two recognition networks are obtained through mutual supervision training, which improves the accuracy and robustness of recognition.
[0155] According to an embodiment of the present application, the present application also provides an electronic device, a readable storage medium and a computer program product.
[0156] Figure 7A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.
[0157] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 702 or a computer program loaded from a storage unit 708 to a RAM (Random Access Memory) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An I / O (Input / Output) interface 705 is also connected to the bus 704.
[0158] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0159] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Units), various dedicated AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the training method of the image recognition model. For example, in some embodiments, the training method of the image recognition model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the training method of the image recognition model described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute the training method of the image recognition model in any other appropriate manner (for example, by means of firmware).
[0160] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application Specific Standard Products), SOCs (System On Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that may be executed and / or interpreted on a programmable system including at least one programmable processor that may be a dedicated or general-purpose programmable processor that may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0161] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0162] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only-Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0163] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0164] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), the Internet, and blockchain networks.
[0165] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services (Virtual Private Server). The server may also be a server of a distributed system, or a server combined with a blockchain.
[0166] According to an embodiment of the present application, the present application also provides a computer program product, which, when an instruction processor in the computer program product executes, executes the training method of the image recognition model proposed in the above embodiment of the present application.
[0167] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.
[0168] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.
Claims
1. A method for training an image recognition model, comprising: Acquire a training data set, wherein the training data set includes a first image set, a second image set, and a labeling result corresponding to each first image in the first image set, wherein each second image in the second image set is generated after the corresponding first image is subjected to a first transformation; Input any first image and the corresponding second image into the first encoder and the second encoder in the initial twin network respectively to obtain a first feature map corresponding to the any first image and a second feature map corresponding to the corresponding second image; Inputting the first feature map and the second feature map into the first decoder and the second decoder in the initial twin network respectively to obtain a first prediction result and a second prediction result; According to the second feature map, the first feature map, the first prediction result, the second prediction result and the annotation result corresponding to any one of the first images, the initial twin network is modified to obtain a first image recognition network and a second image recognition network; The step of modifying the initial twin network to obtain a first image recognition network and a second image recognition network includes: Performing a second transformation on the second feature map to obtain a third feature map, wherein the second transformation and the first transformation are inverse transformations of each other; determining a first difference between the third characteristic map and the first characteristic map; Determining a second difference between the first prediction result and the annotation result corresponding to any one of the first images; Determining a third difference between the second prediction result and the annotation result corresponding to any one of the first images; fusing the first difference, the second difference, and the third difference to determine a loss value; When the loss value is greater than a threshold, the initial twin network is modified based on the loss value to obtain a first image recognition network and a second image recognition network.
2. The method of claim 1, wherein: The step of inputting any one of the first images and the corresponding second image into a first encoder and a second encoder in the initial twin network respectively comprises: Determine a first position vector corresponding to the first image according to a first coordinate value of each first pixel point in any one of the first images; Determining a second position vector corresponding to the second image according to a second coordinate value of each second pixel point in the corresponding second image; The any one first image and the first position vector are input into the first encoder, and the corresponding second image and the second position vector are input into the second encoder.
3. The method of claim 1, wherein: The obtaining of the first prediction result and the second prediction result includes: Obtaining first probabilities that any of the first images belongs to each type of image, and first size deviation values corresponding to the first images when the first images belong to each type of image; Obtaining second probabilities that the corresponding second images belong to each type of image, and second size deviation values corresponding to the second images when the corresponding second images belong to each type of image; Determining a first target type to which any one of the first images belongs according to each of the first probabilities; Determining a first size of an object in any of the first images according to a first size deviation value corresponding to when any of the first images belongs to the first target type and a first reference size corresponding to the first target type; Determining, according to each of the second probabilities, a second target type to which the corresponding second image belongs; The second size of the object in the corresponding second image is determined according to the second size deviation value corresponding to when the corresponding second image belongs to the second target type and the second reference size corresponding to the second target type.
4. The method of claim 3, wherein: The annotation result includes the size and type of the object in each first image, and the method further includes: Determine, according to the type of the object in each first image, a sub-image set corresponding to each type of image, wherein the type of objects in each first image in each sub-image set is the same; According to the size of the objects in each first image in each sub-image set, a reference size corresponding to each type of image is determined.
5. The method according to any one of claims 1 to 2, wherein: The annotation result includes at least one of the following: the size of the two-dimensional detection frame of the object, the size of the object, the three-dimensional center point of the object, the orientation angle of the object, and the type of the object.
6. A training device for an image recognition model, comprising: A first acquisition module is used to acquire a training data set, wherein the training data set includes a first image set, a second image set, and a labeling result corresponding to each first image in the first image set, wherein each second image in the second image set is generated after the corresponding first image is subjected to a first transformation; A second acquisition module is used to input any first image and the corresponding second image into the first encoder and the second encoder in the initial twin network respectively, so as to obtain a first feature map corresponding to the any first image and a second feature map corresponding to the corresponding second image; A third acquisition module, used to input the first feature map and the second feature map into the first decoder and the second decoder in the initial twin network respectively, so as to obtain a first prediction result and a second prediction result; A training module, used to modify the initial twin network according to the second feature map, the first feature map, the first prediction result, the second prediction result and the annotation result corresponding to any one of the first images, so as to obtain a first image recognition network and a second image recognition network; The training module comprises: a transformation unit, configured to perform a second transformation on the second feature map to obtain a third feature map, wherein the second transformation and the first transformation are inverse transformations of each other; a first determining unit, configured to determine a first difference between the third characteristic map and the first characteristic map; A second determining unit, configured to determine a second difference between the first prediction result and a labeling result corresponding to any one of the first images; A third determining unit, configured to determine a third difference between the second prediction result and a labeling result corresponding to any one of the first images; A training unit, configured to modify the initial twin network based on the first difference, the second difference, and the third difference to obtain a first image recognition network and a second image recognition network; The training unit is used to: fusing the first difference, the second difference, and the third difference to determine a loss value; When the loss value is greater than a threshold, the initial twin network is modified based on the loss value to obtain a first image recognition network and a second image recognition network.
7. The device according to claim 6, wherein: The second acquisition module is used to: Determine a first position vector corresponding to the first image according to a first coordinate value of each first pixel point in any one of the first images; Determining a second position vector corresponding to the second image according to a second coordinate value of each second pixel point in the corresponding second image; The any one first image and the first position vector are input into the first encoder, and the corresponding second image and the second position vector are input into the second encoder.
8. The device of claim 6, wherein: The third acquisition module is used to: Obtaining first probabilities that any of the first images belongs to each type of image, and first size deviation values corresponding to the first images when the first images belong to each type of image; Obtaining second probabilities that the corresponding second images belong to each type of image, and second size deviation values corresponding to the second images when the corresponding second images belong to each type of image; Determining a first target type to which any one of the first images belongs according to each of the first probabilities; Determining a first size of an object in any of the first images according to a first size deviation value corresponding to when any of the first images belongs to the first target type and a first reference size corresponding to the first target type; Determining, according to each of the second probabilities, a second target type to which the corresponding second image belongs; The second size of the object in the corresponding second image is determined according to the second size deviation value corresponding to when the corresponding second image belongs to the second target type and the second reference size corresponding to the second target type.
9. The device of claim 8, wherein: The annotation result includes the size and type of the object in each first image, and the device further includes: A first determination module is used to determine a sub-image set corresponding to each type of image according to the type of the object in each first image, wherein the type of the object in each first image in each sub-image set is the same; The second determination module is used to determine the reference size corresponding to each type of image according to the size of the objects in each first image in each sub-image set.
10. The device according to any one of claims 6 to 7, wherein: The annotation result includes at least one of the following: the size of the two-dimensional detection frame of the object, the size of the object, the three-dimensional center point of the object, the orientation angle of the object, and the type of the object.
11. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.
12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-5.
13. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 5.
Citation Information
Patent Citations
Image feature extraction method and device, training method and device, electronic equipment and medium
CN112580666A
Difference information acquisition method, related device and computer program product
CN112990046A
YOLO-based image target recognition method and apparatus, electronic device, and storage medium
WO2020164282A1
