Model training method, traffic marking recognition method, device, equipment and medium
By introducing centroid loss values into machine learning model training, the problem of low training efficiency is solved, and the model recognition accuracy is improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-17
- Publication Date
- 2026-03-24
AI Technical Summary
In existing technologies, machine learning model training is guided solely by network loss, resulting in low training efficiency and low recognition accuracy.
By introducing centroid loss values, multiple first images related to the input image are identified in the model to be trained, and features at different levels are extracted. The centroid parameters and centroid loss values of each image are calculated, and the model is trained by combining the recognition results.
It improves the efficiency and accuracy of model training, and the centroid loss value can more accurately guide model training and promote rapid model convergence.
Smart Images
Figure CN116311126B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of data recognition, and in particular to a model training method, a traffic marking recognition method, device, equipment and medium. BACKGROUND
[0002] In the field of data recognition, a machine learning model is often used for target recognition. For example, in the intelligent driving scenario, a trained machine learning model can be used to recognize traffic markings in a road image.
[0003] In related technologies, the model loss used for training the model is determined only according to the network loss of the model to be trained. Since the network loss is difficult to accurately guide the model training, the model training process is inefficient, and the model recognition accuracy is low. SUMMARY
[0004] To solve or partially solve the problems in the related art, the present application provides a model training method, a traffic marking recognition method, device, equipment and medium, which can accelerate the model training process by introducing the centroid loss value, improve the model training efficiency and accuracy.
[0005] The first aspect of the present application provides a model training method, which comprises:
[0006] determining a plurality of first images related to an input image to be recognized in a model to be trained;
[0007] determining a second image corresponding to each first image after different level feature extraction of each first image;
[0008] determining a centroid parameter of each first image and a centroid parameter of each second image;
[0009] determining a centroid loss value according to the centroid parameter of each first image and the centroid parameter of the second image corresponding to each first image;
[0010] training the model to be trained based on at least the centroid loss value and the recognition result of the image to be recognized.
[0011] According to the model training method provided by the present application, the determination of a plurality of first images related to an input image to be recognized in a model to be trained comprises:
[0012] performing a plurality of first preprocessing operations on the image to be recognized to determine a first image corresponding to each first preprocessing operation, thereby obtaining a plurality of first images.
[0013] According to the model training method provided by the present application, the model to be trained comprises a plurality of convolution layers and a plurality of deconvolution layers, and the convolution layers and the deconvolution layers correspond one by one.
[0014] Each of the convolutional layers is configured to extract semantic features corresponding to a current level from a received image, and output a first feature map containing the semantic features to a next convolutional layer connected to the current convolutional layer and a deconvolutional layer corresponding to the current convolutional layer.
[0015] Each of the deconvolutional layers is configured to fuse the first feature map output by the convolutional layer corresponding to a current level and a received image to obtain a second feature map, and pass the second feature map to a next deconvolutional layer connected to the current deconvolutional layer.
[0016] According to the model training method provided in the present application, the determination of the second images corresponding to the different levels of feature extraction of each of the first images comprises:
[0017] The second feature maps output by at least some of the deconvolutional layers are taken as the second images.
[0018] The at least some of the deconvolutional layers include a deconvolutional layer used to generate a recognition result of the to-be-recognized image.
[0019] According to the model training method provided in the present application, the determination of the center-of-mass parameters of each of the first images comprises:
[0020] Each of the first images is divided into a plurality of image blocks.
[0021] The center of mass of each of the image blocks is determined.
[0022] The center-of-mass parameters of each of the first images are determined according to the centers of mass of all the image blocks corresponding to each of the first images.
[0023] According to the model training method provided in the present application, the division of each of the first images into a plurality of image blocks comprises:
[0024] The image center point of each of the first images is determined.
[0025] A first division line is determined as a vertical line passing through the image center point, and a second division line is determined as a horizontal line passing through the image center point.
[0026] Each of the first images is divided by the first division line and the second division line to obtain a plurality of image blocks corresponding to each of the first images.
[0027] According to the model training method provided in the present application, the determination of the center-of-mass parameters of each of the first images according to the centers of mass of all the image blocks corresponding to each of the first images comprises:
[0028] normalizing the centroid of each image block corresponding to each of the first images to obtain normalized centroids of all the image blocks;
[0029] performing weighted summation on the normalized centroids of all the image blocks to obtain the centroid parameter of each of the first images.
[0030] According to the model training method provided in the present application, the centroid loss value is determined according to the centroid parameter of each of the first images and the centroid parameter of the second image corresponding to each of the first images, and the centroid loss value comprises:
[0031] respectively subtracting the centroid parameter of each of the first images from the centroid parameter of the second image corresponding to each of the first images to obtain a plurality of centroid difference values;
[0032] performing weighted summation on the plurality of centroid difference values to obtain the centroid loss value.
[0033] The second aspect of the present application provides a traffic marking recognition method, which comprises:
[0034] obtaining a road image containing traffic markings;
[0035] inputting the road image into a recognition model to obtain a result image output by the recognition model;
[0036] The result image comprises a recognition result of marking the traffic markings in the road image, and the recognition model is trained based on the model training method described above.
[0037] The third aspect of the present application provides a model training device, which comprises:
[0038] a first processing module configured to determine a plurality of first images related to an input to-be-recognized image in a to-be-trained model;
[0039] a second processing module configured to determine a second image corresponding to each of the first images after different level feature extraction is performed on each of the first images;
[0040] a third processing module configured to determine a centroid parameter of each of the first images and a centroid parameter of each of the second images;
[0041] a fourth processing module configured to determine a centroid loss value according to the centroid parameter of each of the first images and the centroid parameter of the second image corresponding to each of the first images;
[0042] a fifth processing module configured to train the to-be-trained model based on at least the centroid loss value and a recognition result of the to-be-recognized image.
[0043] The fourth aspect of the present application provides a traffic marking recognition device, the device comprises:
[0044] An acquisition module is configured to acquire a to-be-recognized image containing traffic markings.
[0045] An identification module is configured to input the to-be-recognized image into an identification model to obtain a result image output by the identification model.
[0046] The result image includes an identification result of marking traffic markings in the to-be-recognized image, and the identification model is trained based on the model training method described above.
[0047] The fifth aspect of the present application provides an electronic device, comprising:
[0048] A processor; and
[0049] A memory having executable code stored thereon, when the executable code is executed by the processor, the processor executes the method described above.
[0050] The sixth aspect of the present application provides a computer-readable storage medium having executable code stored thereon, when the executable code is executed by the processor of the electronic device, the processor executes the method described above.
[0051] The technical solution provided by the present application can include the following beneficial effects:
[0052] By determining a plurality of first images related to the input to-be-recognized image and a second image corresponding to each of the first images after different level feature extraction is performed on the first image, the centroid parameters of the first image and the second image are calculated, and then the centroid loss value of the to-be-trained model is calculated. Since the centroid loss value can represent the centroid position difference between the input end and the output end in the to-be-trained model at different levels, the introduction of the centroid loss value can accurately guide the model training, improve the model training efficiency, and the model obtained by training has higher recognition accuracy.
[0053] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS
[0054] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings, in which like reference characters designate like elements throughout the several views.
[0055] Figure 1 is a flowchart of the model training method according to an embodiment of the present application;
[0056] Figure 2 is a structural schematic diagram of a model to be trained in an embodiment of the present application;
[0057] Figure 3 is a flow schematic diagram of a traffic marking recognition method shown in an embodiment of the present application;
[0058] Figure 4 is a road image containing a road surface traffic marking in an embodiment of the present application;
[0059] Figure 5 is a recognition marker schematic diagram of a road surface traffic marking in an embodiment of the present application;
[0060] Figure 6 is a recognition result schematic diagram of a road surface traffic marking recognized by a traditional model in an embodiment of the present application;
[0061] Figure 7 is a recognition result schematic diagram of a road surface traffic marking recognized by a recognition model trained in the embodiment in an embodiment of the present application;
[0062] Figure 8 is a structural schematic diagram of a model training apparatus shown in an embodiment of the present application;
[0063] Figure 9 is a structural schematic diagram of a traffic marking recognition apparatus shown in an embodiment of the present application;
[0064] Figure 10 is a structural schematic diagram of an electronic device shown in an embodiment of the present application. DETAILED DESCRIPTION
[0065] Embodiments of the present application will be described in more detail by referring to the drawings. Although embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present application is more thoroughly and completely conveyed to those skilled in the art, and the scope of the present application is fully conveyed to those skilled in the art.
[0066] The terms used in the present application are merely for the purpose of describing particular embodiments and are not intended to limit the present application. As used in the present application and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "and / or," as used herein, refer to and encompass any or all possible combinations of one or more associated listed items.
[0067] It should be understood that although the terms "first," "second," "third," etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.
[0068] This embodiment relates to the field of data recognition and can be specifically applied to target recognition scenarios, such as the recognition of traffic markings in intelligent driving, where target recognition is performed through machine learning.
[0069] In related technologies, because machine learning models rely solely on network loss to guide model parameter adjustments during training, the models struggle to converge quickly, thus affecting training efficiency and resulting in inefficient training processes.
[0070] To address the aforementioned issues, this application provides a model training method that can improve model training efficiency and model recognition accuracy by optimizing the model training process.
[0071] The following combination Figures 1 to 10 This application provides a detailed description of the technical solutions for the model training method, traffic marking recognition method, apparatus, equipment, and medium provided in the embodiments of this application.
[0072] Figure 1 This is a schematic flowchart illustrating the model training method in an embodiment of this application.
[0073] See Figure 1 The model training method provided in this application specifically includes:
[0074] Step 101: Identify multiple first images in the model to be trained that are associated with the input image to be identified.
[0075] In this embodiment, the model to be trained can be a machine learning model built from an untrained deep learning network. The input of the model to be trained is the image to be recognized, and the output is the result image labeled with the target recognition result.
[0076] The image to be identified can be an image containing the target to be identified. For example, in an intelligent driving scenario, the image to be identified can be a road image containing traffic markings. The image to be identified can be obtained by an image acquisition device, such as a camera installed on a vehicle or a roadside camera placed on both sides of the road.
[0077] The first image can be an image obtained after the to-be-identified image is resized, for example, the to-be-identified image can be subjected to a size scaling operation, and thus the first image is obtained.
[0078] Step 102: determining a respective second image corresponding to each first image after different level feature extraction is performed on the first image.
[0079] In this embodiment, the second image can be a feature map containing different level semantic features, and the feature map can be obtained by performing different level feature extraction on the first image, and contains different level semantic features in the first image.
[0080] Step 103: respectively determining a centroid parameter of each first image and a centroid parameter of each second image.
[0081] In this embodiment, the centroid parameter is mainly used to represent the position of the image centroid in the image, and can be presented in the form of a centroid coordinate, and can be obtained by solving the pixel value of at least part of the key pixel points in the image.
[0082] Step 104: determining a centroid loss value according to the centroid parameter of each first image and the centroid parameter of the second image corresponding to each first image.
[0083] In this embodiment, the centroid loss value is mainly used to represent the difference information of the image centroid between the output end and the input end of the to-be-trained model, and can be determined by analyzing the centroid position difference between each first image and the second image corresponding to the first image.
[0084] Step 105: training the to-be-trained model based at least on the centroid loss value and the recognition result of the to-be-identified image.
[0085] In this embodiment, by introducing the centroid loss value, the model training can be constrained. In the model training process, the centroid loss value and the network loss obtained according to the recognition result of the to-be-identified image are both involved in the determination of the model loss, so that the model loss can more accurately guide the model training. In this way, the to-be-trained model can quickly converge, the training efficiency of the to-be-trained model is improved, and the model recognition accuracy is improved.
[0086] In some embodiments, the plurality of first images related to the input to-be-identified image in the to-be-trained model are determined, and the determination specifically includes:
[0087] The to-be-identified image is subjected to a plurality of first preprocessing operations, a first image corresponding to each first preprocessing operation is determined, and a plurality of first images are obtained.
[0088] In this embodiment, the first preprocessing operation can be an image size reduction operation. Specifically, multiple image size reduction operations can be performed on the to-be-identified image to obtain first images of different sizes.
[0089] In actual applications, the reduction ratio can be determined according to the size of the to-be-identified image. If the to-be-identified image is large, the reduction ratio of each reduction operation can be appropriately increased. If the to-be-identified image is small, the reduction ratio of each reduction operation can be appropriately reduced.
[0090] The number of first preprocessing operations can be determined according to the number of base layers in the to-be-trained model, that is, the number of first preprocessing operations can be determined according to the number of feature extraction levels. For example, if the to-be-trained model includes four base layers, the to-be-identified image can be subjected to four first preprocessing operations. Since the output end of the to-be-trained model includes four sub-base layers, if the feature map output by each sub-base layer of the output end is used as a second image, at most four second images can be determined. Therefore, the number of first preprocessing operations determined by the above method can be an upper limit of the number of first preprocessing operations.
[0091] By performing multiple first preprocessing operations on the to-be-identified image, multiple first images related to the to-be-identified image can be obtained. The first images are more convenient to obtain, and more data can be provided for subsequent determination of the centroid loss value.
[0092] In some embodiments, the to-be-trained model specifically includes multiple convolution layers and multiple deconvolution layers, and the convolution layers and the deconvolution layers correspond to each other in one-to-one correspondence.
[0093] Each convolution layer is configured to extract semantic features corresponding to the current level from a received image, and output a first feature map containing the semantic features to a next convolution layer connected to the current convolution layer and a deconvolution layer corresponding to the current convolution layer.
[0094] Each deconvolution layer is configured to fuse the first feature map output by the convolution layer corresponding to the current level and the received image to obtain a second feature map, and pass the second feature map to a next deconvolution layer connected to the current deconvolution layer.
[0095] The embodiment provides a specific structure scheme of the to-be-trained model. In the embodiment, the to-be-trained model can be divided into an input end and an output end. The input end is composed of multiple convolution layers. The convolution layers are sequentially connected, that is, the input end of any convolution layer is connected with the output end of the previous convolution layer, and the output end of any convolution layer is connected with the input end of the next convolution layer, so that the semantic features extracted by the convolution layers can be transmitted level by level until the last convolution layer, and the acquisition function of the multi-level semantic features in the to-be-recognized image is realized.
[0096] The output end is composed of multiple deconvolution layers. The deconvolution layers are sequentially connected, that is, the input end of any deconvolution layer is connected with the output end of the previous deconvolution layer, and the output end of any deconvolution layer is connected with the input end of the next deconvolution layer. In addition, each convolution layer is connected with a deconvolution layer, that is, each sub-basic layer of the output end corresponds to each sub-basic layer of the input end.
[0097] Figure 2 An example of a specific structure of the to-be-trained model is shown. Referring to FIG. 2, Figure 2 the to-be-trained model specifically includes an encoder 201 and a decoder 202. The encoder 201 can be understood as the input end of the to-be-trained model, and the decoder 202 can be understood as the output end of the to-be-trained model.
[0098] The encoder 201 includes four convolution layers, that is, a first convolution layer, a second convolution layer, a third convolution layer and a fourth convolution layer. The first convolution layer, the second convolution layer, the third convolution layer and the fourth convolution layer are sequentially connected.
[0099] The first convolution layer is used for extracting semantic features of the to-be-recognized image at a first level, and outputting a first feature map containing the semantic features at the first level to the second convolution layer and the decoder 202 respectively.
[0100] The second convolution layer is used for extracting semantic features of the to-be-recognized image at a second level based on the semantic features at the first level, and outputting a first feature map containing the semantic features at the second level to the third convolution layer and the decoder 202 respectively.
[0101] The third convolution layer is used for extracting semantic features of the to-be-recognized image at a third level based on the semantic features at the second level, and outputting a first feature map containing the semantic features at the third level to the fourth convolution layer and the decoder 202 respectively.
[0102] The fourth convolution layer is used for extracting semantic features of the to-be-recognized image at a fourth level based on the semantic features at the third level, and outputting a first feature map containing the semantic features at the fourth level to the decoder 202.
[0103] The embodiment is directed to a case where the encoder 201 comprises four convolutional layers, as shown in Figure 2 The network architecture of the entire model to be trained can be roughly divided into four basic layers, namely Figure 2 The four basic layers can be divided into two parts, the encoder 201 and the decoder 202, and the encoder 201 and the decoder 202 can be connected through a splicing channel, Figure 2 The kernel size of the network of the target recognition model shown in
[0104] As shown in Figure 2 The encoder 201 is sequentially connected with four convolutional layers, and each convolutional layer is defined according to the data transmission direction, that is, the first convolutional layer, the second convolutional layer, the third convolutional layer and the fourth convolutional layer correspond to Figure 2 The basic layers in the encoder 201 from top to bottom in
[0105] On the one hand, each convolutional layer will perform a down-sampling operation on the input image, so that the channel number of the input image is doubled, and the size of the image (specifically the length and width of the image) will be reduced to half of the original size, for example, the size of the input image is 480x800, and after one down-sampling operation, the size of the image is reduced to 240x400; on the other hand, each convolutional layer will perform convolution operation, normalization operation and activation operation on the input image to extract the semantic features of the input image at the current level, and the semantic features of multiple levels are extracted through multiple convolutional layers.
[0106] Referring to Figure 2 The number of layers of the decoder 202 in the embodiment is the same as that of the encoder 201, and in the scenario shown in Figure 2 The decoder 202 comprises four deconvolutional layers connected in sequence, and the four deconvolutional layers of the decoder 202 are sequentially defined according to the flow direction of data, specifically, the first deconvolutional layer, the second deconvolutional layer, the third deconvolutional layer and the fourth deconvolutional layer correspond to Figure 2 The basic layers in the decoder 202 from bottom to top in
[0107] In the case where the encoder 201 comprises four convolutional layers, the decoder 202 comprises a first deconvolutional layer, a second deconvolutional layer, a third deconvolutional layer and a fourth deconvolutional layer, and the first deconvolutional layer, the second deconvolutional layer, the third deconvolutional layer and the fourth deconvolutional layer are sequentially connected;
[0108] The first deconvolutional layer is configured to receive a first feature map containing semantic features of a fourth level output by the fourth convolutional layer in the encoder, take the first feature map as a second feature map of a first level, and pass the obtained second feature map to a second deconvolutional layer;
[0109] The second deconvolutional layer is configured to fuse the first feature map containing semantic features of a third level output by the third convolutional layer in the encoder and the second feature map output by the first deconvolutional layer, and pass a second feature map of a second level obtained to a third deconvolutional layer;
[0110] The third deconvolutional layer is configured to fuse the first feature map containing semantic features of a second level output by the second convolutional layer in the encoder and the second feature map output by the second deconvolutional layer, and pass a second feature map of a third level obtained to a fourth deconvolutional layer;
[0111] The fourth deconvolutional layer is configured to fuse the first feature map containing semantic features of a first level output by the first convolutional layer in the encoder and the second feature map output by the third deconvolutional layer, and obtain a result image.
[0112] In one aspect, each deconvolutional layer can perform an upsampling operation on the input feature map. The upsampling operation can reduce the channel number of the input feature map by one and increase the size (specifically, the length and width of the image) of the feature map by one, for example, the image size of the input feature map is 480x800, and after one upsampling operation, the image size is increased to 960x1600.
[0113] On the other hand, the deconvolutional layer can fuse the received image with the first feature map of the current level output by the encoder, so as to extract more detailed semantic features.
[0114] It can be understood that the image received by the deconvolutional layer in the embodiment can be the feature map passed by the previous deconvolutional layer connected to the current deconvolutional layer, or can be other feature maps except the first feature map output by the convolutional layer corresponding to the current level, for example, the first feature map output by the convolutional layer of the previous level.
[0115] In some embodiments, determining the respective second images of each first image after feature extraction of different levels, specifically comprising:
[0116] Taking the second feature maps output by at least part of the deconvolutional layers in the multiple deconvolutional layers as the multiple second images;
[0117] The at least part of the deconvolutional layers includes the deconvolutional layers used to generate the recognition result of the to-be-recognized image.
[0118] In the embodiment, the to-be-trained model is Figure 2As an example of the structure shown, the feature map obtained after different level feature extraction of the first image can be taken as the second image, for example, the first feature map obtained after first level feature extraction of the first image through a convolution layer is input into the deconvolution layer corresponding to the convolution layer, and the second feature map can be obtained after feature fusion by the deconvolution layer, which is a second image. In this way, at least part of the second feature map output by the deconvolution layer can be taken as the second image.
[0119] Since the recognition result of the to-be-recognized image output by the to-be-trained model, i.e., the result image, is a more representative image at the output end of the to-be-trained model, the result image needs to be included in the plurality of second images as much as possible, that is, at least part of the deconvolution layer includes a deconvolution layer for generating the recognition result of the to-be-recognized image.
[0120] In some embodiments, the second image can also be obtained by performing a plurality of second preprocessing operations on the result image. The second preprocessing operation can be a size enlargement operation on the result image, for example, performing a plurality of size enlargement operations on the result image by a certain ratio, thereby obtaining a plurality of second images.
[0121] In this embodiment, regardless of which determination method of the second image is used, it is necessary to ensure that the first image and the second image are in one-to-one correspondence. The correspondence relationship can be artificially set or can be determined according to the level structure of the to-be-trained model. Specifically, it can be reasonably set according to actual application needs, and will not be described in detail here.
[0122] In some embodiments, the center of mass parameter of each first image is determined, specifically including:
[0123] Each first image is divided into a plurality of image blocks;
[0124] The center of mass of each image block is determined;
[0125] The center of mass parameter of each first image is determined according to the centers of mass of all image blocks corresponding to each first image.
[0126] In this embodiment, the center of mass of each image block can be obtained by the coordinate solving method of the image center of mass, which is also called the image barycenter. The image center of mass can be determined by the pixel value of each image point in the image. For a two-dimensional image, the center of mass coordinates can be independently solved in the x direction and the y direction, that is, for the center of mass in the x direction, the sum of the pixels on the left and right sides of the image at the center of mass is equal; for the center of mass in the y direction, the sum of the pixels on the upper and lower sides of the image at the center of mass is equal.
[0127] Suppose there are n image points in the image, and the coordinate of any image point in the x direction is x i The corresponding pixel value is p iIf the centroid has coordinate x in the x-direction, then the following relationship exists:
[0128] (1)
[0129] According to the above formula (1), we can obtain:
[0130] (2)
[0131] Similar to the calculation method described above, the coordinates y of the centroid in the y-direction can be obtained, thus yielding the centroid coordinates of the two-dimensional image.
[0132] By dividing the first image into multiple image blocks, obtaining the centroid of each image block, and finally summing them up to solve the centroid parameters of the first image, this method of determining centroid parameters is more accurate than the method of directly solving the centroid of the first image to determine the centroid parameters, thereby improving the reliability of the centroid loss value.
[0133] In practical applications, there are various ways to divide the first image into blocks. The rationality of the division of multiple image blocks will directly affect the accuracy of the centroid parameter solution. Therefore, this embodiment provides a relatively reasonable image block division scheme, as detailed in the following embodiment.
[0134] In some embodiments, each first image is divided into multiple image blocks, specifically including:
[0135] Determine the center point of each first image;
[0136] The first and second dividing lines are determined by taking the vertical line passing through the center point of the image as the first dividing line and the horizontal line passing through the center point of the image as the second dividing line.
[0137] Each first image is segmented using the first and second dividing lines to obtain multiple image blocks corresponding to each first image.
[0138] In this embodiment, the first image is divided into multiple image blocks by uniformly dividing the first image. Specifically, the center point of the first image can be determined first, and then the first dividing line and the second dividing line can be determined by the vertical and horizontal straight lines passing through the center point of the image. The first image is divided according to the first dividing line and the second dividing line to obtain four equally divided image blocks. In this case, the first image can be divided into a "field" shape to achieve uniform division of the image blocks.
[0139] In some embodiments, if the first image is a rectangular image, the first image can also be divided into four image blocks by taking two diagonal lines of the first image as the first and second dividing lines. This kind of division scheme can also divide the first image into four image blocks, but the four image blocks obtained by division are all triangles and the specific shapes are slightly different. For scenes with slightly lower accuracy requirements, this kind of division scheme can be used.
[0140] Similarly, the second image can also adopt any of the above image block division schemes. In actual application, the image block division schemes of the first image and the second image need to be consistent as much as possible, so as to facilitate unified solving calculation and other data processing operations in the subsequent solving link.
[0141] In some embodiments, the centroid parameter of each first image is determined according to the centroids of all image blocks corresponding to each first image, specifically including:
[0142] The centroids of all image blocks corresponding to each first image are normalized to obtain normalized centroids of all image blocks.
[0143] The normalized centroids of all image blocks are weighted and summed to obtain the centroid parameter of each first image.
[0144] In this embodiment, in order to facilitate unified solving calculation of the centroids of all image blocks, the centroids of all image blocks can be uniformly normalized before solving to obtain normalized centroids of all image blocks, that is, the centroid coordinates of all image blocks are normalized to a smaller data range, and then the normalized centroids are weighted and summed. Specifically, the coordinates of each coordinate axis corresponding to the normalized centroid of each image block are added to obtain a sub-centroid parameter of the image block, and then the sub-centroid parameters are weighted and summed, that is, the sub-centroid parameter of each image block is multiplied by the weight corresponding to the image block to obtain the centroid parameter of the first image.
[0145] It should be noted that in the process of weighting and summing the normalized centroids of all image blocks in this embodiment, the weight values of the image blocks can be fixed values or dynamically changing values. In this embodiment, the weight values of the image blocks are dynamically changing values, that is, dynamic weight values.
[0146] In actual application, the initial values of the dynamic weight values corresponding to the image blocks are consistent, for example, they can all be 0.25. In the training process of the to-be-trained model, a loss value is generated for each image block, and the dynamic weight value is updated in the loss iteration process, so that the centroid parameter of each first image is constantly updated in the iteration process, thereby improving the reliability of the centroid parameter.
[0147] Similarly, the centroid parameter of the second image can also be determined by using the same calculation method as that of the centroid parameter of the first image.
[0148] In some embodiments, the centroid loss value is determined according to the centroid parameter of each first image and the centroid parameter of the second image corresponding to each first image, specifically comprising:
[0149] The centroid parameter of each first image is subtracted from the centroid parameter of the second image corresponding to each first image respectively to obtain a plurality of centroid difference values;
[0150] The plurality of centroid difference values are weighted and summed to obtain the centroid loss value.
[0151] In this embodiment, the centroid difference value can represent the difference in centroid position between the first image and the second image corresponding to the first image. After the plurality of centroid difference values are weighted and summed, the centroid loss value can represent the comprehensive difference in centroid position between the plurality of first images and the second images corresponding to the first images.
[0152] It can be understood that in the process of weighting and summing the plurality of centroid difference values, the weight value of each centroid difference value can be obtained by averaging a plurality of weight values related to the centroid difference value. Specifically, the plurality of weight values related to the centroid difference value can include the weight values of each image block in the first image corresponding to the centroid difference value and the weight values of each image block in the second image corresponding to the centroid difference value.
[0153] In practical applications, the centroid loss value can be solved by cooperating a plurality of centroid solving modules and a summing module, as shown in Figure 3 When the first image and the second image are both four, four centroid solving modules and one summing module can be set, and the four centroid solving modules are the first centroid solving module 203, the second centroid solving module 204, the third centroid solving module 205 and the fourth centroid solving module 206.
[0154] The one end of the first centroid solving module 203 is used to receive the first image of the first level obtained by the first first pre-processing operation on the image to be recognized in the encoder 201, and the other end of the first centroid solving module 203 is used to receive the feature map output by the fourth deconvolution layer in the decoder 202, i.e. the second image of the fourth level in the decoder 202. The first centroid solving module 203 is used to solve the centroid parameters of the received first image and second image, calculate the difference between the centroid parameters of the first image and the second image, obtain the first centroid difference value, and transmit the first centroid difference value to the summing module 207.
[0155] One end of the second centroid solving module 204 is configured to receive the second-level first image obtained after the second first preprocessing operation of the image to be recognized in the encoder 201, and the other end of the second centroid solving module 204 is configured to receive the feature map output by the third deconvolution layer in the decoder 202, i.e., the second image of the third level in the decoder 202. The second centroid solving module 204 is configured to solve the centroid parameters of the received first image and second image, calculate the difference between the centroid parameters of the first image and the second image, obtain the second centroid difference value, and transmit the second centroid difference value to the summation module 207.
[0156] One end of the third centroid solving module 205 is configured to receive the third-level first image obtained after the third first preprocessing operation of the image to be recognized in the encoder 201, and the other end of the third centroid solving module 205 is configured to receive the feature map output by the second deconvolution layer in the decoder 202, i.e., the second image of the second level in the decoder 202. The third centroid solving module 205 is configured to solve the centroid parameters of the received first image and second image, calculate the difference between the centroid parameters of the first image and the second image, obtain the third centroid difference value, and transmit the third centroid difference value to the summation module 207.
[0157] One end of the fourth centroid solving module 206 is configured to receive the fourth-level first image obtained after the fourth first preprocessing operation of the image to be recognized in the encoder 201, and the other end of the fourth centroid solving module 206 is configured to receive the feature map output by the first deconvolution layer in the decoder 202, i.e., the second image of the first level in the decoder 202. The fourth centroid solving module 206 is configured to solve the centroid parameters of the received first image and second image, calculate the difference between the centroid parameters of the first image and the second image, obtain the fourth centroid difference value, and transmit the fourth centroid difference value to the summation module 207.
[0158] The summation module 207 is mainly configured to receive the first centroid difference value, the second centroid difference value, the third centroid difference value, and the fourth centroid difference value, and perform weighted summation on the above four centroid difference values to obtain a centroid loss value.
[0159] In some embodiments, the training of the to-be-trained model is performed based on at least the centroid loss value and the recognition result of the image to be recognized, and specifically includes:
[0160] Based on the recognition result of the image to be recognized and the label information of the image to be recognized, a network loss value of the to-be-trained model is determined.
[0161] The centroid loss value and the network loss value are weighted and summed to obtain a model loss value.
[0162] The to-be-trained model is trained by using the model loss value.
[0163] In this embodiment, the model loss value can guide the to-be-trained model to adjust the model parameters until the model loss value reaches the preset threshold, the model converges, and the optimal model parameters are determined, that is, the trained recognition model is obtained.
[0164] In one specific embodiment, the training process of the to-be-trained model will also involve the following process:
[0165] Firstly, the obtained to-be-identified image sample and label data are combined to generate sample data required for training;
[0166] Secondly, the data that does not meet the standard in the sample data is normalized to obtain sample data that meets the standard. Specifically, the data that does not correspond to the to-be-identified image sample in the label data can be modified so that the label data corresponds to the to-be-identified image sample, thereby realizing the normalization of the sample data. This step can also be omitted in a scenario where the training accuracy requirement is not high.
[0167] Thirdly, the sample data is divided into test data and training data by random grouping, and the test data and the training data are saved in a preset sample database, for example, an MBD (Model Based Definition, Model Based Definition) database.
[0168] Fourthly, the training data in the sample database is read, and the read training data is parsed, for example, into a matrix form, for example, a 480x800x3 matrix. The parsed data is input into the to-be-trained model built by the deep learning network for training to obtain a trained recognition model.
[0169] Fifthly, the test data is used to test the trained recognition model, and the test result is compared with the real label data. When the comparison result meets the test passing requirement, a test-passing recognition model is obtained.
[0170] It should be noted that the training method of the recognition model provided in the embodiments of the present application is mainly applied to the training link of the fourth step. In the model training process, the introduction of the centroid loss value determined in the embodiments can guide the recognition model to quickly complete the training task, thereby improving the efficiency and accuracy of the model training.
[0171] Figure 3 is a flowchart of the traffic marking recognition method shown in the embodiments of the present application.
[0172] Referring to Figure 4 The traffic marking recognition method provided in the embodiments of the present application specifically includes:
[0173] Step 301: Obtain a road image containing traffic markings. Step 302: Extract the traffic markings from the road image to obtain a to-be-identified image sample;
[0174] Step 302: inputting the road image into the recognition model to obtain a result image output by the recognition model;
[0175] The result image includes a recognition result of marking the traffic marking in the road image; and the recognition model is trained based on the model training method.
[0176] In this embodiment, the recognition model trained by the above embodiment is mainly applied to a traffic marking recognition scene in the field of intelligent driving. In this scene, the to-be-recognized image is a road image containing traffic markings, and the traffic marking refers to a mark on the road surface that uses lines, arrows, characters, vertical surface marks, protruding road signs, and contour marks to transmit guidance, restrictions, warnings, and other traffic information to traffic participants.
[0177] In actual application, the execution subject of the target object recognition method can be a processor or a server. For example, in the automatic driving scene, the execution subject of the target object recognition method can be a processor deployed on a vehicle or a processor or server deployed outside the vehicle.
[0178] The road image can be collected by an image collection device installed on the vehicle. For example, the road image in front of the vehicle can be collected by a camera installed at the front end of the vehicle. The result image can be an image in which the contour of the traffic marking is marked in the to-be-recognized image.
[0179] In order to verify the recognition accuracy improvement effect of the traffic marking recognition method provided in this embodiment, the same to-be-recognized image is recognized by using a conventional model trained based on only a network loss and a recognition model trained by introducing a centroid loss value provided in this embodiment.
[0180] In the comparative verification, the to-be-recognized image is a road image containing traffic markings, Figure 5 An example of the road image is shown. The road image is collected by a camera installed in front of the vehicle. The road image contains vehicles on the road, road signs on both sides of the road, traffic lights, trees, and traffic markings on the road surface. The recognition target of this embodiment is to recognize the position of the traffic marking on the road surface from the to-be-recognized road image. Specifically, four traffic markings can be visible on the road surface in the to-be-recognized image. The recognition label of the road surface traffic marking corresponding to the to-be-recognized image can be seen in Figure 6 The recognition result of the road surface traffic marking obtained by recognizing the to-be-recognized image by using the conventional model can be seen in Figure 7 The recognition result of the road surface traffic marking obtained by recognizing the to-be-recognized image by using the recognition model trained in this embodiment can be seen in Figure 6 .
[0181] willFigure 7 the recognition result of the road surface traffic marking shown in FIG. 6, Figure 5 the recognition result of the road surface traffic marking shown in FIG. 6 respectively with Figure 6 It can be found that the recognition result of the road surface traffic marking shown in FIG. 6 is compared with Figure 7 the recognition result of the road surface traffic marking shown in FIG. 6 has a missed recognition situation on the traffic marking on the road surface far away, and the recognition effect is poor, while Figure 5 the recognition result of the road surface traffic marking shown in FIG. 6 is more close to Figure 8 the recognition label of the road surface traffic marking shown in FIG. 6, and the recognition effect is better.
[0182] Therefore, compared with the traditional model trained without introducing the centroid loss value, the recognition model trained by the model training method provided in the embodiment has more optimal model parameters after training, the recognition accuracy of the traffic marking by the recognition model is effectively improved, and the model training efficiency is also improved.
[0183] Corresponding to the foregoing application function implementation method embodiment, the present application also provides a model training device, a traffic marking recognition device, an electronic device and corresponding embodiments.
[0184] Figure 8 is a structural schematic diagram of the model training device provided in the embodiment of the present application.
[0185] Referring to Figure 9 , the model training device provided in the embodiment of the present application specifically comprises:
[0186] The first processing module 401 is configured to determine a plurality of first images related to the input to-be-recognized image in the to-be-trained model.
[0187] The second processing module 402 is configured to determine a respective second image corresponding to each first image after different level feature extraction is performed on the first image.
[0188] The third processing module 403 is configured to respectively determine a centroid parameter of each first image and a centroid parameter of each second image.
[0189] The fourth processing module 404 is configured to determine a centroid loss value according to the centroid parameter of each first image and the centroid parameter of the respective second image corresponding to each first image.
[0190] The fifth processing module 405 is configured to train the to-be-trained model based on at least the centroid loss value and the recognition result of the to-be-recognized image.
[0191] In some embodiments, the first processing module 401 can be specifically configured to:
[0192] The plurality of first images are obtained by performing a plurality of first preprocessing operations on the to-be-identified image, and determining a first image corresponding to each first preprocessing operation.
[0193] In some embodiments, the to-be-trained model can specifically include a plurality of convolutional layers and a plurality of deconvolutional layers, and the convolutional layers and the deconvolutional layers correspond to each other in one-to-one correspondence.
[0194] Each convolutional layer is configured to extract semantic features corresponding to a current level of the received image, and output a first feature map containing the semantic features to a next convolutional layer connected to the current convolutional layer and a deconvolutional layer corresponding to the current convolutional layer.
[0195] Each deconvolutional layer is configured to fuse the first feature map output by the received image and the convolutional layer corresponding to the current level, to obtain a second feature map, and pass the second feature map to a next deconvolutional layer connected to the current deconvolutional layer.
[0196] In an example embodiment, the second processing module 402 can be specifically configured to:
[0197] output the second feature map output by at least part of the plurality of deconvolutional layers as a plurality of second images.
[0198] The at least part of the deconvolutional layers includes a deconvolutional layer used to generate an identification result of the to-be-identified image.
[0199] In some embodiments, the fourth processing module 404 can specifically determine the center of mass parameter of each first image by the following process:
[0200] Divide each first image into a plurality of image blocks, respectively.
[0201] Determine the center of mass of each image block, respectively.
[0202] Determine the center of mass parameter of each first image according to the centers of mass of all image blocks corresponding to the first image.
[0203] In an example embodiment, the fourth processing module 404 can specifically divide each first image into a plurality of image blocks by the following process:
[0204] Determine an image center point of each first image.
[0205] Determine a first division line as a vertical line passing through the image center point, and a second division line as a horizontal line passing through the image center point.
[0206] Divide each first image by the first division line and the second division line to obtain a plurality of image blocks corresponding to the first image.
[0207] In the example embodiment, the fourth processing module 404 can specifically determine the centroid parameter of each first image according to the centroids of all image blocks corresponding to each first image by the following process:
[0208] normalizing the centroids of all image blocks corresponding to each first image to obtain normalized centroids of all image blocks;
[0209] performing weighted summation on the normalized centroids of all image blocks to obtain the centroid parameter of each first image.
[0210] In some embodiments, the fourth processing module 404 can be specifically configured to:
[0211] respectively subtract the centroid parameter of each first image from the centroid parameter of the second image corresponding to each first image to obtain a plurality of centroid difference values;
[0212] performing weighted summation on the plurality of centroid difference values to obtain a centroid loss value.
[0213] In summary, the model training apparatus provided in the embodiment introduces a centroid loss value in the model training link, the centroid loss value can represent the centroid position difference between the input end and the output end at different levels in the to-be-trained model, and the introduction of the centroid loss value can accurately guide the model training, improve the model training efficiency, and obtain a model with higher recognition accuracy.
[0214] Figure 9 is a structural schematic diagram of a traffic marking recognition apparatus shown in an embodiment of the present application.
[0215] Referring to Figure 10 , the traffic marking recognition apparatus provided in the embodiment of the present application specifically comprises:
[0216] an acquisition module 501 configured to acquire a to-be-recognized image containing traffic markings;
[0217] an identification module 502 configured to input the to-be-recognized image into an identification model to obtain a result image output by the identification model;
[0218] wherein the result image comprises an identification result of marking the traffic markings in the to-be-recognized image; and the identification model is trained based on the above model training method.
[0219] As to the apparatus in the above embodiments, the specific manners in which various modules perform operations have been described in detail in the embodiments about the method, and will not be described in detail here.
[0220] Figure 10 is a structural schematic diagram of an electronic device shown in an embodiment of the present application.
[0221] Referring to The electronic device 600 includes a memory 601 and a processor 602.
[0222] The processor 602 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can be any conventional processor.
[0223] The memory 601 can include various types of storage units, such as a system memory, a read-only memory (ROM), and a permanent storage device. Among them, the ROM can store static data or instructions required by the processor 602 or other modules of the computer. The permanent storage device can be a read-write storage device. The permanent storage device can be a non-volatile storage device that does not lose stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a mass storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device can be a removable storage device (such as a floppy disk, an optical drive). The system memory can be a read-write storage device or a volatile read-write storage device, such as a dynamic random access memory. The system memory can store some or all of the instructions and data required by the processor during runtime.
[0224] In addition, the memory 601 can include a combination of any computer readable storage media, including various types of semiconductor storage chips (such as DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), magnetic disks and / or optical disks. In some embodiments, the memory 601 can include a read and / or write removable storage device, such as a compact disc (CD), a read-only digital versatile disc (such as DVD-ROM, double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (such as an SD card, a min SD card, a Micro-SD card, etc.), a magnetic floppy disk, etc. The computer readable storage medium does not include a carrier wave and a transient electronic signal transmitted through a wireless or wired transmission.
[0225] The memory 601 stores executable code, which, when processed by the processor 602, can cause the processor 602 to perform part or all of the above-mentioned methods.
[0226] Furthermore, the method according to the present application can also be implemented as a computer program or a computer program product, which comprises computer program code instructions for executing some or all of the steps of the above-mentioned method according to the present application.
[0227] Alternatively, the present application can also be implemented as a computer readable storage medium (or a non-transitory machine readable storage medium or a machine readable storage medium) having stored thereon executable codes (or computer programs or computer instruction codes) which, when executed by a processor of an electronic device (or a server, etc.), cause the processor to perform some or all of the steps of the above-mentioned method according to the present application.
[0228] The above has described the embodiments of the present application, the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes are obvious to those skilled in the art without departing from the scope and spirit of the described embodiments. The selection of the terms used herein is intended to best explain the principles of the embodiments, practical application or improvement of the technology in the market, or to enable other ordinary skilled in the art to understand the embodiments disclosed herein.
Claims
1. A model training method, characterized in that, include: Identify multiple first images in the model to be trained that are related to the input image to be identified; Determine the corresponding second image after performing feature extraction at different levels on each of the first images; The centroid parameters of each first image and each second image are determined respectively. The centroid loss value is determined based on the centroid parameters of each first image and the centroid parameters of the corresponding second image for each first image. The training model is performed based at least on the centroid loss value and the recognition result of the image to be recognized; wherein, During model training, the centroid loss value and the network loss obtained based on the recognition result of the image to be recognized both participate in the determination of the model loss. The centroid parameter of the first image is obtained by solving for the pixel values of the pixels in the first image, and the centroid parameter of the second image is obtained by solving for the pixel values of the pixels in the second image.
2. The model training method according to claim 1, characterized in that, The determination of multiple first images related to the input image to be identified in the model to be trained includes: The image to be identified is subjected to multiple first preprocessing operations to determine the first image corresponding to each first preprocessing operation, thereby obtaining multiple first images.
3. The model training method according to claim 1, characterized in that, The model to be trained includes multiple convolutional layers and multiple deconvolutional layers, with each convolutional layer corresponding to one of the deconvolutional layers. Each convolutional layer is used to extract the semantic features of the received image at the current level, and output a first feature map containing the semantic features to the next convolutional layer connected to the current convolutional layer and the deconvolutional layer corresponding to the current convolutional layer; Each deconvolutional layer is used to fuse the received image with the first feature map output by the convolutional layer corresponding to the current layer to obtain a second feature map, and then pass the second feature map to the next deconvolutional layer connected to the current deconvolutional layer.
4. The model training method according to claim 3, characterized in that, The step of determining the second image corresponding to each of the first images after performing feature extraction at different levels includes: The second feature map output by at least a portion of the deconvolution layers in the multi-layer deconvolution layer is used as a plurality of second images; The at least partial deconvolution layer includes a deconvolution layer for generating the recognition result of the image to be recognized.
5. The model training method according to claim 1, characterized in that, Determining the centroid parameters of each of the first images includes: Each of the first images is divided into multiple image blocks; Determine the centroid of each of the aforementioned image blocks; The centroid parameter of each first image is determined based on the centroids of all image blocks corresponding to each first image.
6. The model training method according to claim 5, characterized in that, The step of dividing each of the first images into multiple image blocks includes: Determine the image center point of each of the first images; The first dividing line and the second dividing line are determined by taking the vertical straight line passing through the center point of the image as the first dividing line and the horizontal straight line passing through the center point of the image as the second dividing line. Each first image is segmented using the first segmentation line and the second segmentation line to obtain multiple image blocks corresponding to each first image.
7. The model training method according to claim 5, characterized in that, The step of determining the centroid parameter of each first image based on the centroids of all image blocks corresponding to each first image includes: Normalize the centroids of all image blocks corresponding to each first image to obtain the normalized centroids of all image blocks; The centroid parameters of each of the first image are obtained by weighted summation of the normalized centroids of all image blocks.
8. The model training method according to claim 1, characterized in that, The step of determining the centroid loss value based on the centroid parameters of each first image and the centroid parameters of the corresponding second image for each first image includes: The centroid parameter of each first image is subtracted from the centroid parameter of the corresponding second image for each first image to obtain multiple centroid difference values; The weighted sum of the multiple centroid differences yields the centroid loss value.
9. A method for recognizing traffic markings, characterized in that, include: Acquire road images containing traffic markings; The road image is input into the recognition model to obtain the result image output by the recognition model; The resulting image includes the recognition results of marking traffic markings in the road image; the recognition model is trained based on the model training method described in any one of claims 1 to 8.
10. A model training device, characterized in that, include: The first processing module is used to determine multiple first images in the model to be trained that are related to the input image to be recognized. The second processing module is used to determine the second image corresponding to each of the first images after performing feature extraction at different levels on each first image. The third processing module is used to determine the centroid parameter of each first image and the centroid parameter of each second image respectively. The centroid parameter of the first image is obtained by solving the pixel value of the first image, and the centroid parameter of the second image is obtained by solving the pixel value of the second image. The fourth processing module is used to determine the centroid loss value based on the centroid parameters of each of the first images and the centroid parameters of the corresponding second images of each of the first images. The fifth processing module is used to train the model to be trained based at least on the centroid loss value and the recognition result of the image to be recognized. During the model training process, the centroid loss value and the network loss obtained based on the recognition result of the image to be recognized both participate in the determination of the model loss.
11. A traffic marking recognition device, characterized in that, include: The acquisition module is used to acquire images containing traffic markings to be recognized; The recognition module is used to input the image to be recognized into the recognition model and obtain the result image output by the recognition model; The resulting image includes the recognition result of marking traffic markings in the image to be recognized; the recognition model is trained based on the model training method described in any one of claims 1 to 8.
12. An electronic device, characterized in that, include: processor; as well as A memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method as described in any one of claims 1-8.
13. A computer-readable storage medium, characterized in that, It stores executable code that, when executed by a processor of an electronic device, causes the processor to perform the method as described in any one of claims 1-8.
Citation Information
Patent Citations
Method and device for automatically adjusting and aligning pictures based on deep learning
CN113837998A
Lane line detection method and device based on improved U-Net network
CN114821510A