Image recognition model training method and system and image recognition method

Abstract

The present invention discloses an image recognition model training method, system, and image recognition method. The training method comprises: collecting a large number of images related to road scenes; the images include at least vehicles, pedestrians, buildings, ground road markings, roadside traffic signs, signs in the space above the road, and obstacles located on the road; each image is labeled with a corresponding frame and preprocessed; the preprocessed images are divided into a training set and a validation set according to a preset ratio; and a constructed YOLOv4 network model is iteratively trained using the training set and validation set until the loss function converges. The trained optimal YOLOv4 network model is obtained as the image recognition model to realize road scene image recognition. Compared with traditional CNN models, this method can reduce the computational complexity without causing a loss in accuracy, and has faster detection speed and stronger real-time performance.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to an image recognition model training method and system and an image recognition method. Background Art

[0002] Currently, deep learning is about learning the inherent patterns and representational hierarchies of sample data. The information gained from this learning process is highly helpful in interpreting data such as text, images, and sound. Its ultimate goal is to enable machines to acquire the same analytical and learning capabilities as humans, enabling them to recognize data such as text, images, and sound. Deep learning is a complex machine learning algorithm that has achieved results in speech and image recognition that far surpass previous technologies.

[0003] Deep learning has achieved significant results in search technology, data mining, machine learning, machine translation, natural language processing, multimedia learning, speech recognition, recommendation and personalization technologies, and other related fields. Deep learning enables machines to mimic human activities such as seeing, hearing, and thinking, solving many complex pattern recognition challenges and significantly advancing AI-related technologies.

[0004] Another technology, autonomous driving, has also become a hot topic of research. Autonomous driving systems utilize advanced communications, computing, networking, and control technologies to achieve real-time, continuous control of trains. Modern communication methods, directly connected to the train, enable two-way data communication between trains and the ground, with fast transmission rates and large amounts of information. This allows tracking trains and control centers to instantly determine the exact location of the preceding train, making operational management more flexible, control more efficient, and more adaptable to the demands of autonomous driving. Autonomous driving systems are integrated systems that integrate numerous advanced technologies. Key components, such as environmental information acquisition and intelligent decision-making and control, rely on innovative and breakthrough technologies in sensor technology, image recognition, electronics, computing, and control. The long-term development of autonomous vehicles relies on breakthroughs and innovations in multiple technologies.

[0005] Therefore, integrating deep learning image recognition technology with unmanned driving technology to achieve accurate road judgment through accurate image recognition as a means to assist unmanned driving has become a new research object.

[0006] Currently, there is no good and accurate method for image recognition in the field of unmanned driving. Summary of the Invention

[0007] In view of the above problems, the present invention provides an image recognition model training method and system and an image recognition method that solve at least some of the above technical problems. The method requires less computational effort for training and has high recognition accuracy.

[0008] In a first aspect, an embodiment of the present invention provides an image recognition model training method, comprising:

[0009] S10. Collect a large number of images related to road scenes; the images include at least: vehicles, pedestrians, buildings, ground road markings, roadside traffic signs, signs in the space above the road, and obstacles on the road;

[0010] S20, each of the pictures is marked with a corresponding frame, and the pictures are pre-processed;

[0011] S30, dividing the pre-processed images into a training set and a validation set according to a preset ratio;

[0012] S40. Using the training set and the validation set, the constructed YOLOv4 network model is iteratively trained until the loss function converges, and the optimal YOLOv4 network model after training is obtained as the image recognition model to realize road scene image recognition.

[0013] In one embodiment, each of the pictures has a corresponding annotation frame, including:

[0014] The images were annotated into VOC format using Labelimg software, and vehicles, pedestrians, buildings, ground road markings, roadside traffic signs, signs in the space above the road, and obstacles on the road in each image were annotated.

[0015] In one embodiment, preprocessing the image includes:

[0016] Use the Mosaic data augmentation method to increase the number of samples.

[0017] In one embodiment, the YOLOv4 network model adopts a lightweight MobileNetv3 network, and introduces a depth-separable convolution and inception network structure into the detector.

[0018] In a second aspect, an embodiment of the present invention provides an image recognition model training system, comprising:

[0019] A collection module is used to collect a large number of pictures related to road scenes; the pictures include at least: vehicles, pedestrians, buildings, ground road markings, roadside traffic signs, signs in the space above the road, and obstacles on the road;

[0020] A preprocessing module, used for preprocessing the pictures; and each picture is provided with a corresponding annotation frame;

[0021] A construction module is used to divide the pre-processed images into a training set and a validation set according to a preset ratio;

[0022] The training module uses the training set and the validation set to iteratively train the constructed YOLOv4 network model until the loss function converges, and obtains the optimal YOLOv4 network model after training as the image recognition model to realize road scene image recognition.

[0023] In a third aspect, an embodiment of the present invention further provides an image recognition method, comprising:

[0024] While the vehicle is moving forward, it acquires images of the road scene ahead in real time;

[0025] Inputting the front road scene image into an image recognition model; the image recognition model is trained by the image recognition model training method described in the above embodiment;

[0026] The image recognition model outputs a recognition result of the front road scene image.

[0027] The beneficial effects of the above technical solutions provided by the embodiments of the present invention include at least:

[0028] An embodiment of the present invention provides a method for training an image recognition model, comprising: collecting a large number of images related to road scenes; the images include at least vehicles, pedestrians, buildings, ground road markings, roadside traffic signs, signs in the space above the road, and obstacles located on the road; each image is labeled with a corresponding box and the images are preprocessed; the preprocessed images are divided into a training set and a validation set according to a preset ratio; and a constructed YOLOv4 network model is iteratively trained using the training set and validation set until the loss function converges, obtaining the optimal trained YOLOv4 network model as the image recognition model to implement road scene image recognition. Compared with traditional CNN models, this method can reduce computational complexity without compromising accuracy, and has faster detection speed and greater real-time performance.

[0029] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purposes and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

[0030] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0032] Figure 1 A flowchart of an image recognition model training method provided by an embodiment of the present invention;

[0033] Figure 2 A structural diagram of the YOLOv4 network model provided in an embodiment of the present invention;

[0034] Figure 3 A diagram of the MobileNetv3 network structure provided by an embodiment of the present invention;

[0035] Figure 4a This is a diagram of the Inception network structure provided by an embodiment of the present invention;

[0036] Figure 4b This is a diagram of the adjusted Inception network structure provided by an embodiment of the present invention;

[0037] Figure 5 This is a block diagram of the image recognition model training system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0038] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0039] Example 1:

[0040] Reference Figure 1 As shown, an embodiment of the present invention provides an image recognition model training method, comprising:

[0041] S10. Collect a large number of images related to road scenes; the images include at least: vehicles, pedestrians, buildings, ground road markings, roadside traffic signs, signs in the space above the road, and obstacles on the road;

[0042] S20, each of the pictures is marked with a corresponding frame, and the pictures are pre-processed;

[0043] S30, dividing the pre-processed images into a training set and a validation set according to a preset ratio;

[0044] S40. Iteratively train the constructed YOLOv4 network model using the training set and validation set until the loss function converges, obtaining the optimal trained YOLOv4 network model as the image recognition model to implement road scene image recognition. Compared with traditional CNN models, this method can reduce the computational complexity without sacrificing accuracy, and has faster detection speed and greater real-time performance.

[0045] In one embodiment, each of the images in step S20 is provided with a corresponding annotation frame, and the image can be annotated using the annotation tool labelimg to generate an XML annotation file. For example, approximately 500,000 video frame images can be randomly numbered and given a reasonable serial number, such as 0000000 to 0499999; labelimg software is used to annotate data, and each image name corresponds to an XML annotation file with a corresponding name, such as image 0000000.jpg, and the annotation file is 0000000.xml. The scope of annotation includes: image location, image name (such as 0000000.jpg), image width and height, image dimensions, the name of the annotated pre-selected frame, and the coordinate value of the pre-selected frame; the types of annotation include: vehicles, pedestrians, buildings, ground road markings, roadside traffic signs, signs in the space above the road, and obstacles located on the road, such as overpasses, or objects that obstruct traffic (objects that appear abnormally on the road). The image data can be divided into training data set, test data set and verification data set, of which the training data set accounts for 70%, the test data set accounts for 20%, and the verification data set accounts for 10%.

[0046] In one embodiment, the pre-processing of the image in step S20 includes: using Mosaic data enhancement to increase the number of samples.

[0047] For example: photometric distortion: adjusting brightness, contrast, hue, saturation, and noise; geometric distortion: random scaling, shearing, flipping, rotation, and style transfer; datasets can also be constructed by random erasing and filling with random or complementary zero values; for example, multiple rectangular areas in an image can be randomly or uniformly selected and replaced with zeros to achieve data augmentation on the feature map.

[0048] You can also stitch multiple images together to make the network training more effective and increase data diversity.

[0049] Increase the number of targets and the parameters of BN that can count multiple images at once: If using the BN layer, try to set the batch_size as large as possible, because the BN layer mainly requires the mean and variance. The larger the batch_size, the closer it is to the mean and variance of the entire dataset, and the better the effect. However, due to limited video memory, batch_size cannot be increased indefinitely, but it is possible to stitch n images together, which is equivalent to parallel input of the original image with batch_size = n. For example, Mosasic uses four images to enrich the background of the detected object. And during the BN calculation, the data of four images is calculated at once. Implementation method: Read four images each time, flip, scale, change the color gamut, etc., and arrange them in four directions to combine the images and frames.

[0050] In one embodiment, Figure 2 As shown, the YOLOv4 object detection network uses the CSPDarknet53 as its backbone and comprises five CSP modules. The convolution kernel size before each module is 3×3 with a stride of 2, further enhancing the network's learning capabilities. A Path Aggregation Network (PANet) serves as the backbone, supplemented by a Spatial Pyramid Pooling (SPP) module. This employs 1×1, 5×5, 9×9, and 13×13 max pooling schemes to increase the receptive field and isolate more important contextual features. The YOLOv3 detection head is retained as the head. Finally, a fully connected layer outputs the training results, including bounding box regression coordinates, object classification results, and confidence scores. The output of each YOLOv4 layer is shown below. The explanation of how each layer is obtained is annotated after each row; unannotated layers represent convolutions on the feature maps in the previous row. The YOLOv4 network has 161 layers and consumes a total of 128.46 BFLOPS at a 608×608 resolution, compared to 141 BFLOPS for YOLOv3.

[0051] In this embodiment, in order to further reduce the GPU memory and model storage requirements of the YOLOv4 network model, the feature extraction network of YOLOv4 is changed to the MobileNetv3 network structure. Figure 3As shown in the figure, a Sequential Encoding (SE) structure is added to the BottleNet architecture and placed after the depthwise filter. Because the SE structure consumes a certain amount of time, reducing the number of channels in the expansion layer of the SE-containing architecture to 1 / 4 improves accuracy without increasing the processing time. This reduces the model's parameters while maintaining a certain level of accuracy. Secondly, to further reduce the model weight, depthwise separable convolutions are used in the CBL_block1 and CBL_block2 modules of the detection network instead of traditional convolutions, avoiding the memory shortages and high latency associated with complex models. Finally, the last Conv2D 3×3 convolution layer at each scale in the original YOLOv4 network is replaced with an Inception network structure to improve the accuracy of obstacle detection in images.

[0052] Specifically, in order to achieve lightweight and miniaturized image recognition models, this embodiment uses the MobileNetV3 network as the backbone feature extraction network to replace the original CSPDarkNet-53 network in YOLOv4, so as to reduce the overall computational complexity and memory usage of the network.

[0053] In order to further make the network model lightweight and improve the real-time performance of the algorithm. In this embodiment, depth-wise separable convolution is introduced in the detection network instead of traditional convolution to reduce the model's requirements for GPU memory and model storage. In the depth-wise separable network, the convolution operation can be divided into two steps. Assuming that the size of the convolution kernel is Dk×Dk, the size of the input feature map is DF×DF, M and N represent the number of input and output channels, the total amount of calculation of the traditional convolution is F1, and the total amount of calculation of the depth-wise separable convolution is F2. The ratio of the two calculation amounts F2 / F1 can be described by the following formula.

[0054]

[0055] The CBL_block1 structure before and after the original SPP structure consists of two 1×1 convolutional layers and one 3×3 convolutional layer. In the PANet network, the CBL_block2 structure immediately following the concat operation consists of three 1×1 convolutional layers and two 3×3 convolutional layers. Drawing on the concept of depthwise separable convolution, this embodiment replaces the original 3×3 traditional convolution with 1×1 point convolution and 3×3 depthwise convolution. Furthermore, the nonlinear h-swish function, which performs better in deep models, is used as the activation function, replacing the leaky ReLU function used in the original detection network. Compared to the original detection network, this improved detection network not only speeds up computation but also increases network depth and nonlinearity.

[0056] The Inception network structure maintains good computing performance while maintaining network sparsity, such as Figure 4a As shown in the figure, using convolution kernels of different sizes means receptive fields of different sizes, and the final splicing means the fusion of features of different scales. The reason why the convolution kernel sizes are 1, 3, and 5 is mainly to facilitate alignment. Pooling is also embedded in Inception. The further the network goes, the more abstract the features are, and the receptive field involved in each feature is also larger. Therefore, as the number of layers increases, the ratio of 3x3 and 5x5 convolutions should also increase.

[0057] In order to further optimize the training speed, the size of the layer can be adjusted to balance the computational load between the subnets of each model. Use 1x1 convolution kernel for dimensionality reduction, refer to Figure 4b As shown in Figure 1, a 1x1 convolution kernel is added before the 3x3 convolution and 5x5 convolution layers, respectively; and a 1x1 convolution kernel is added after the 3x3 maximum pooling layer.

[0058] For example, if the output of the previous layer is 100x100x128, after passing through a 5x5 convolutional layer with 256 outputs (stride = 1, pad = 2), the output data size is 100x100x256. The convolutional layer parameters are 5x5x128x256. If the output of the previous layer first passes through a 1x1 convolutional layer with 32 outputs, and then through a 5x5 convolutional layer with 256 outputs, the final output data size is still 100x100x256, but the number of convolutional parameters has been reduced to 1x1x128x32 + 5x5x32x256, a reduction of approximately 4 times.

[0059] In the Inception structure, a large number of 1x1 matrices are used, mainly for two purposes: 1) to reduce the dimensionality of the data; 2) to introduce more nonlinearities and improve generalization ability, because the convolution must pass through the ReLU activation function.

[0060] Example 2:

[0061] Based on the same inventive concept, an embodiment of the present invention also provides an image recognition model training system. Since the principle of solving the problem by this system is similar to that of the aforementioned method, the implementation of this system can refer to the implementation of the aforementioned method, and the repeated parts will not be repeated.

[0062] Reference Figure 5 As shown, an image recognition model training system includes:

[0063] A collection module is used to collect a large number of pictures related to road scenes; the pictures include at least: vehicles, pedestrians, buildings, ground road markings, roadside traffic signs, signs in the space above the road, and obstacles on the road;

[0064] A preprocessing module, used for preprocessing the pictures; and each picture is provided with a corresponding annotation frame;

[0065] A construction module is used to divide the pre-processed images into a training set and a validation set according to a preset ratio;

[0066] The training module uses the training set and the validation set to iteratively train the constructed YOLOv4 network model until the loss function converges, and obtains the optimal YOLOv4 network model after training as the image recognition model to realize road scene image recognition.

[0067] In the pre-processing module, each picture is marked with a corresponding box, including:

[0068] The images were annotated into VOC format using Labelimg software, and vehicles, pedestrians, buildings, ground road markings, roadside traffic signs, signs in the space above the road, and obstacles on the road in each image were annotated.

[0069] The image is preprocessed, including: using Mosaic data enhancement to increase the number of samples.

[0070] The YOLOv4 network model in the training module adopts the lightweight MobileNetv3 network, and introduces deep separable convolution and inception network structures in the detector.

[0071] Example 3:

[0072] An embodiment of the present invention further provides an image recognition method, comprising:

[0073] (1) The vehicle acquires the image of the road ahead in real time while driving forward;

[0074] (2) inputting the front road scene image into an image recognition model; the image recognition model is trained by the image recognition model training method described in the above embodiment 1;

[0075] (3) Outputting the recognition result of the front road scene image through the image recognition model.

[0076] By adopting the image recognition method of this embodiment, it can be combined with lidar point cloud processing to better assist autonomous driving and provide a reference for intelligent decision-making in autonomous driving.

[0077] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A method for training an image recognition model, characterized in that: include: S10, collecting a large number of pictures related to road scenes; The image at least includes: vehicles, pedestrians, buildings, ground road markings, roadside traffic signs, signs in the space above the road, and obstacles on the road; S20, each of the pictures is marked with a corresponding frame, and the pictures are pre-processed; Preprocessing includes: Use the Mosaic data augmentation method to increase the number of samples; S30, dividing the pre-processed images into a training data set accounting for 70%, a test data set accounting for 20%, and a validation data set accounting for 10%; S40. Use the training data set, test data set and validation data set to iteratively train the constructed YOLOv4 network model until the loss function converges, and obtain the optimal YOLOv4 network model after training as an image recognition model to realize road scene image recognition; wherein, the feature extraction network of YOLOv4 is changed to the MobileNetv3 network structure, the SE structure is added to the bottlenet structure, and is placed after the depthwise filter; depthwise separable convolution is used instead of traditional convolution in the CBL_block1 and CBL_block2 modules of the detector, and the inception network structure is introduced in the last layer of Conv2D 3×3 convolution at each scale; and the h-swish function is used as the activation function.

2. The image recognition model training method according to claim 1, characterized in that: Each of the pictures is accompanied by a corresponding annotation box, including: The images were annotated into VOC format using Labelimg software, and vehicles, pedestrians, buildings, ground road markings, roadside traffic signs, signs in the space above the road, and obstacles on the road in each image were annotated.

3. An image recognition model training system, characterized in that: Using the image recognition model training method according to any one of claims 1-2, the system comprises: A collection module is used to collect a large number of pictures related to road scenes; the pictures include at least: vehicles, pedestrians, buildings, ground road markings, roadside traffic signs, signs in the space above the road, and obstacles on the road; A preprocessing module, used for preprocessing the pictures; and each picture is provided with a corresponding annotation frame; A construction module is used to divide the pre-processed images into a training data set accounting for 70%, a test data set accounting for 20%, and a validation data set accounting for 10%; The training module uses the training data set, test data set, and validation data set to iteratively train the constructed YOLOv4 network model until the loss function converges, and obtains the optimal trained YOLOv4 network model as the image recognition model to realize road scene image recognition. In this module, the feature extraction network of YOLOv4 is changed to the MobileNetv3 network structure, the SE structure is added to the bottlenet structure, and is placed after the depthwise filter; depthwise separable convolution is used instead of traditional convolution in the CBL_block1 and CBL_block2 modules of the detector, and the inception network structure is introduced in the last Conv2D 3×3 convolution layer at each scale; and the h-swish function is used as the activation function.

4. An image recognition method, characterized in that: include: While the vehicle is moving forward, it acquires images of the road scene ahead in real time; Inputting the front road scene image into an image recognition model; The image recognition model is obtained by training the image recognition model training method according to any one of claims 1 to 2; The image recognition model outputs a recognition result of the front road scene image.

Images