Image processing multi-task model, processing method and training method

Abstract

The invention relates to an image processing multi-task model, a processing method and a training method, and the image processing multi-task model comprises a backbone network which is used for carrying out the batch processing of continuous video images in a drive test video, and adding the time sequence information of the video images in the batch processing process of the continuous video images, and obtaining an output feature; the first branch network is used for obtaining a first target detection feature, a second target detection feature and a third target detection feature; the second branch network is used for fusing the deep semantic features output by the optimized deep semantic perception modules to obtain a first fusion feature, and correspondingly processing the first fusion feature to obtain a lane line segmentation result; and the output features of the backbone network are processed by the pooling network and the third branch network to obtain an image quality classification result. Therefore, the system architecture can be simplified, the calculation overhead can be reduced, the overall processing efficiency can be improved, and the false detection rate can be effectively reduced.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing multi-task model, processing method and training method. Background Art

[0002] Target detection technology, image quality recognition technology, and image segmentation and recognition technology play an important role in vehicle driving. In the existing technology, target detection technology can be used to identify vehicles, pedestrians and other targets in the road test video, and image quality recognition technology can be used to identify the clarity, brightness, contrast and other indicators of each frame of the video image in the road test video to ensure the best visual experience. Image segmentation and recognition technology can be used to segment different areas in the video image, and further analyze and process them, so that they can be used for lane line segmentation and recognition.

[0003] At present, target detection technology, image quality recognition technology and image segmentation recognition technology often use single-task models. Each model can only handle a specific task. However, in multi-task scenarios, multiple models are required to work together, which increases the complexity and computational overhead of the system. In addition, due to the neglect of certain factors during processing, it is easy to misdetect or miss detection. Summary of the invention

[0004] The embodiments of the present application provide an image processing multi-task model, processing method and training method, which can simplify the system architecture, reduce computing overhead, improve overall processing efficiency, and effectively reduce false detection rate and misdetection rate.

[0005] In a first aspect, the present application provides an image processing multi-task model, comprising: a backbone network, a pooling network, a first branch network, a second branch network, and a third branch network;

[0006] The backbone network is used to perform batch processing on continuous video images in the drive test video, and to add time sequence information of the video images during the batch processing of the continuous video images to obtain output features;

[0007] The first branch network includes three parallel sub-networks, namely a first sub-network, a second sub-network and a third sub-network;

[0008] The output features of the backbone network are processed by the pooling network and the first sub-network to obtain a first target detection feature;

[0009] The output features of the backbone network are respectively input into the second sub-network and the third sub-network to obtain corresponding second target detection features and third target detection features;

[0010] The sub-network head includes an optimized deep semantic perception module; the optimized deep semantic perception module is used to obtain deep semantic features reflecting target features through multiple different types of attention mechanisms and fusion mechanisms;

[0011] The second branch network is used to fuse the deep semantic features output by each of the optimized deep semantic perception modules to obtain a first fused feature, and process the first fused feature accordingly to obtain a lane segmentation result;

[0012] The output features of the backbone network are processed by the pooling network and the third branch network to obtain an image quality classification result.

[0013] In a second aspect, an embodiment of the present application provides a processing method based on an image processing multi-task model, comprising:

[0014] Acquire continuous video images of the drive test video;

[0015] The continuous video images are input into the image processing multi-task model provided in the embodiment of the present application to obtain a first target detection result, a second target detection result, a third target detection result, a lane line detection result and an image quality classification result.

[0016] In a third aspect, an embodiment of the present application provides a method for training an image processing multi-task model, including:

[0017] Obtaining training samples of the image processing multi-task model, and annotating continuous video images in the training samples;

[0018] The image processing multi-task model is trained using labeled continuous video images to obtain a trained image processing multi-task model; wherein the image processing multi-task model is the image processing multi-task model provided in the embodiment of the present application.

[0019] The above technical solution provided by the embodiment of the present application has the following advantages compared with the prior art:

[0020] The technical solution provided by the embodiment of the present application performs batch processing on continuous images in the road test video through a backbone network, and adds the timing information of the video images during the batch processing to obtain output features, and obtains the first target detection features through the output features of the backbone network through a pooling network and a first sub-network in the first branch network, and obtains the corresponding second target detection features and third target detection features through the second sub-network and the third sub-network of the first branch network respectively; the deep semantic features output by each optimized deep semantic module in the first branch network are fused and processed through the second branch network to obtain a lane line segmentation result, and the features processed by the pooling network are processed through the third branch network to obtain an image quality classification result;

[0021] That is, the multi-task model of the present application can realize shared underlying feature extraction through the backbone network, reduce repeated calculations, reduce computing overhead, and improve the utilization of computing resources; through three branch networks, the three tasks of target detection, lane line segmentation, and image quality recognition can be realized simultaneously. This collaborative processing method not only simplifies the system architecture and reduces hardware resource requirements, but also improves the overall processing efficiency; the multi-task model of the present application, by introducing the timing information of the video image, optimizes the single-image non-timing model in the prior art into a multi-image timing reasoning model, and considers the timing information of the previous and subsequent video images, which can reduce the false detection rate and the false detection rate; by optimizing the deep semantic perception module, deeper and richer deep semantic features can be captured to improve the detection accuracy; by sharing the underlying feature extraction part through the backbone network, information complementarity and collaborative optimization are achieved among multiple tasks, thereby improving the performance of each task. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0024] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0025] Figure 1 It is a structural diagram of a multi-task model for image processing;

[0026] Figure 2 It is a structural diagram of a multi-task model for image processing;

[0027] Figure 3 It is a schematic diagram of the structure of the C3T module;

[0028] Figure 4 It is a schematic diagram of the structure of the SODSS module;

[0029] Figure 5 It is a schematic diagram of the structure of the SS2D module;

[0030] Figure 6 It is a schematic diagram of the structure of a local space block;

[0031] Figure 7 It is a schematic diagram of the structure of the residual gating block;

[0032] Figure 8 It is a flow chart of a processing method based on an image processing multi-task model provided in an embodiment of the present application;

[0033] Fig. 9 It is a schematic diagram of quality classification of video images;

[0034] Fig.10 It is a flow chart of a training method for an image processing multi-task model provided in an embodiment of the present application;

[0035] Fig.11 It is a structural block diagram of a processing device based on an image processing multi-task model provided in an embodiment of the present application;

[0036] Fig.12 It is a structural block diagram of a training device for an image processing multi-task model provided in an embodiment of the present application;

[0037] Fig.13 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0039] The disclosure below provides many different embodiments or examples to implement different structures of the present invention. In order to simplify the disclosure of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present invention. In addition, the present invention can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0040] In order to solve the technical problems in the prior art of increased system complexity and computational overhead, as well as easy false detection or missed detection caused by the collaborative work of multiple models, the present application provides an image processing multi-task model, processing method, training method and device, which can simplify the system architecture, reduce computational overhead, improve overall processing efficiency, and effectively reduce the false detection rate and false detection rate.

[0041] Figure 1 It is a structural diagram of a multi-task model for image processing, such as Figure 1 As shown, it includes: a trunk network 11, a pooling network 12, a first branch network 13, a second branch network 14 and a third branch network 15.

[0042] The backbone network 11 is used to batch process the continuous video images in the drive test video, and add the timing information of the video images during the batch processing of the continuous video images to obtain output features; wherein the timing information of the video images may be the time information corresponding to the video images, and the number of batch processed images may be 8 or may be any number greater than two. The deep information of the video images is shared through the backbone network.

[0043] Among them, the first branch network 13 includes three parallel sub-networks, namely the first sub-network 131, the second sub-network 132 and the third sub-network 133; the output features of the backbone network 11 are processed by the pooling network 12 and the first sub-network 131 to obtain the first target detection features; the output features of the backbone network 11 are respectively input into the second sub-network 132 and the third sub-network 133 to obtain the corresponding second target detection features and third target detection features; wherein the sub-network head includes an optimized deep semantic perception (S Optimized Deep Semantic Sensing, SODSS) module; the optimized deep semantic perception module is used to obtain the deep semantic features reflecting the target through multiple different types of attention mechanisms and fusion mechanisms. Optionally, the first target detection feature, the second target detection feature and the third target detection feature can be used to judge pedestrians, cars and two or three-wheeled vehicles respectively. Among them, the target detection feature is also used to judge driver fatigue or road signs, etc.; that is, through the first sub-network 131, the second sub-network 132 or the third sub-network 133, the driver's fatigue state or road signs can also be detected.

[0044] Among them, the heads of the first sub-network 131, the second sub-network 132 and the third sub-network 133 all include a SODSS module, through which the attention mechanism and the fusion mechanism are repeatedly used in the processing process to obtain the deep semantic features that best reflect the target features. The multiple use of the attention mechanism and the fusion mechanism is conducive to the capture of global features, and the corresponding target detection features can be obtained by performing convolution operations on the deep semantic features. Among them, the target detection features can include the target detection frame information, classification information, and trust information, and the target detection features can also be subsequently processed to obtain the target detection frame and classification information.

[0045] In one embodiment, optionally, the pooling network 12 can be a spatial pyramid pooling (Spatial Pyramid Pooling Fast, SPPF) module, which is an efficient feature extraction module, and its core function is to fuse multi-scale information. The SPPF module achieves the effect of feature pyramid by mapping features of different scales to a feature map of a fixed size, and the output features have the following characteristics: Multi-scale feature fusion: Specifically, the SPPF module processes the input features at different scales through pooling operations, so that the network can capture object features of different sizes. For target detection, it can focus on large and small targets at the same time, thereby improving the robustness of detection. Spatial invariance: Specifically, through pooling operations, the SPPF module reduces the sensitivity in the spatial dimension, so that the network can better extract global context information without being affected by a specific position. This is particularly important for object recognition in complex scenes. Efficient feature extraction: The SPPF module increases the richness of the feature layer in the convolutional neural network, but does not significantly increase the computational cost. It is very critical for real-time detection tasks.

[0046] Among them, the second branch network 14 is used to fuse the deep semantic features output by each optimized deep semantic perception module to obtain a first fused feature, and the first fused feature is processed accordingly to obtain a lane line segmentation result. Among them, the lane line segmentation result may include a lane line segmentation feature, which reflects the lane line information and is mainly used to determine whether the target is in the same lane. In the lane line segmentation task, the resolution requirement is high, and it is necessary to use the fusion of deep semantic features of different resolutions output by multiple SODSS modules to more accurately segment the lane line.

[0047] Among them, the output features of the backbone network 11 are processed by the pooling network 12 and the third branch network 15 to obtain the image quality classification result. Among them, the image quality can be classified into clear, general, fuzzy and meaningless, and background text. In this embodiment, the output features of the backbone network 11 are processed by the pooling network 12 and then processed by the third branch network 15 to obtain the image quality classification result. The main reasons are: 1. Image quality classification does not rely on spatial positioning: Image quality classification mainly evaluates the quality of the overall video image (such as clarity, brightness, contrast, etc.) without caring about the specific position of the object in the video image. Therefore, the pooling operation can well compress the global features into a smaller feature representation, which is conducive to comprehensively judging the overall quality of the video image. 2. Global feature extraction: In image quality classification, pooling operations (especially global average pooling) through the pooling network help generate a global representation of the video image. Through the downsampling of the pooling network, it is possible to focus more on the quality features of the overall video image without having to care about the spatial distribution of details. This is exactly the opposite of the target detection task. Target detection focuses on the location and boundaries of specific objects, while image quality classification focuses on global features. 3. Reduce computational complexity: In image quality classification, the pooling operation of the pooling network can also help reduce the dimension of the feature map, thereby reducing the computational cost and making the classification process more efficient.

[0048] The technical solution provided by the embodiment of the present application performs batch processing on continuous images in the road test video through a backbone network, and adds the timing information of the video images during the batch processing to obtain output features, and obtains the first target detection features through the output features of the backbone network through a pooling network and a first sub-network in the first branch network, and obtains the corresponding second target detection features and third target detection features through the second sub-network and the third sub-network of the first branch network respectively; the deep semantic features output by each optimized deep semantic module in the first branch network are fused and processed through the second branch network to obtain a lane line segmentation result, and the features processed by the pooling network are processed through the third branch network to obtain an image quality classification result;

[0049] That is, the multi-task model of the present application can realize shared underlying feature extraction through the backbone network, reduce repeated calculations, and improve the utilization rate of computing resources; through three branch networks, the three tasks of target detection, lane line segmentation, and image quality recognition can be realized simultaneously. This collaborative processing method not only simplifies the system architecture and reduces the hardware resource requirements, but also improves the overall processing efficiency; the multi-task model of the present application, by introducing the timing information of the video image, optimizes the single-image non-timing model in the prior art into a multi-image timing reasoning model, and takes the timing information into consideration, which can reduce the false detection rate and the false detection rate; by optimizing the deep semantic perception module, deeper and richer deep semantic features can be captured to improve the detection accuracy; by sharing the underlying feature extraction part through the backbone network, information complementarity and collaborative optimization are achieved among multiple tasks, thereby improving the performance of each task.

[0050] In one embodiment, if Figure 2 As shown, the backbone network 11 includes a first convolution module and a target module formed by several target series layers in parallel; wherein, the first convolution module and the target module are connected in series; the target series layer is composed of a first convolution normalization and activation CBS module and a first space and spatiotemporal fusion C3T module connected in series.

[0051] In one embodiment, if Figure 2 As shown, the third branch network 13 includes a second C3T module, an eleventh convolution module and a fully connected module connected in series in sequence; the output features of the pooling network are used as input features of the second C3T module, and the output result of the fully connected module is used as the image quality classification result.

[0052] Among them, Figure 3 As shown, the C3T module includes an upper branch sub-network, a lower branch sub-network and a second CBS module; the upper branch sub-network includes a third CBS module and a bottleneck module (BottleNeck module); the lower branch sub-network includes a fourth CBS module and a first time offset TSM module; wherein the output features of the upper sub-branch network and the output features of the lower branch sub-network are fused as the input features of the second CBS module. The C3T module includes a first C3T module and a second C3T module.

[0053] Among them, in the backbone network 11, the video image can be feature extracted through the first convolution module, and the timing information of the video image can be added, and then processed by 4×(first CBS module+first C3T module) to obtain output features.

[0054] Among them, the features output by 4×(first CBS module + first C3T module) can be understood as the deep feature representation after multi-layer convolution, feature extraction and transformation of the input video image. Specifically: CBS (Convolutional + Softmax) module includes: Convolution block:

[0055] (Convolutional Block): Performs convolution operations on the input video image to extract low-level spatial features, such as edges, textures, etc. These features will gradually transform into more complex features as the number of network layers increases. Softmax: In the context, it may be used to normalize or activate classification outputs to help the network perform classification tasks.

[0056] Among them, the C3T (CSPNet Cross Stage Partial with Transformer-like module) module includes: the CSPNet part reduces the computational complexity by transferring and merging partial features, while maintaining rich gradient information to improve the efficiency and depth of feature expression; the Transformer-like module emphasizes capturing long-range contextual relationships and sequence features, which can help the network capture more global features in the spatial dimension.

[0057] Therefore, the specific output features of 4× (first CBS module + first C3T module) can include deep spatial features: after multiple convolutions (CBS modules), higher-level spatial features can be extracted, including complex shapes, textures, and physical boundary information; multi-scale feature expression: the CSPNet part can capture object features at different levels at multiple scales through staged feature fusion, and all objects from small objects to large objects can be expressed through feature layers; global features: the Transformer-like module part can help capture the long-distance dependency of input features and reflect global context information, so that the network can understand the relationship between objects in the scene, not just local features.

[0058] like Figure 2 As shown, the backbone network includes a first C3T module, and the third branch network includes a second C3T module; the input and output are introduced by taking the second C3T module as an example:

[0059] 1. The output x1(batch,num,20,20,1024) of the SPPF (spatial pyramid pooling) module is the input feature;

[0060] 2. The x1 feature enters both the upper branch sub-network and the lower branch sub-network;

[0061] 3. Among them, the upper branch sub-network is a cascade of the third CBS module (convolution, normalization and activation function series) and the bottleneck module, outputting x2 features;

[0062] 4. Among them, the lower branch sub-network is the fourth CBS (convolution, normalization and activation function series) and the first TSM module, outputting x3 features;

[0063] 5. The next step is to fuse the x2 feature with the x3 feature to obtain the x4 feature;

[0064] 6. Finally, the x4 feature is sent to the second CBS module (convolution, normalization, and activation function series) to obtain the x5 feature with dimension (batch, num, 40, 40, 512).

[0065] Among them, the C3T modules such as the first C3T module and the second C3T module include multiple CBS modules and TSM modules. By combining the CBS modules and the TSM modules, the temporal modeling capability of multi-frame video image features in driving scenes can be significantly improved. The CBS module enhances the importance of key driving scene frames, such as vehicle detection, pedestrian recognition, and traffic sign classification, while the TSM module captures temporal dynamic information in multi-frame sequences through lightweight time offset operations, thereby enhancing the module's understanding and response capabilities to complex driving environments. This combination can not only reduce the risk of overfitting the model on long time series data, but also maintain high computational efficiency and model integration, making real-time image perception and scene understanding more outstanding and reliable.

[0066] In one embodiment, optionally, Figure 2 As shown, in the first branch network 13, the first sub-network 131 also includes a seventh convolution module, which is used to perform a convolution operation on the deep semantic features output by the corresponding optimized deep semantic perception module to obtain a first target detection feature; the second sub-network 132 also includes an eighth convolution module, which is used to perform a convolution operation on the deep semantic features output by the corresponding optimized deep semantic perception module to obtain a second target detection feature; the third sub-network 133 also includes a ninth convolution module, which is used to perform a convolution operation on the deep semantic features output by the corresponding optimized deep semantic perception module to obtain a third target detection feature. Among them, the first target detection feature, the second target detection feature and the third target detection feature can include target detection frame information, classification information, trust, etc. During processing, they can be filtered by trust, and the features with low trust can be filtered. Then, the detection frame, category, etc. of the target detection can be obtained by decoding.

[0067] In one embodiment, if Figure 2 As shown, the second branch network 14 includes a fusion module, a first upsampling module, a tenth convolution module and a second upsampling module connected in series in sequence; wherein the deep semantic features output by each optimized deep semantic module are used as input features of the fusion module, and the output result of the second upsampling module is used as the lane line segmentation result.

[0068] In one embodiment, if Figure 4 As shown, the SODSS module includes: a first series module, a second series module and a third series module; wherein the first series module includes a second convolution module and a first batch normalization module (BatchNorm module) connected in series in sequence; wherein the first batch normalization module includes a first activation function module (Activation module); the specific first activation function module is embedded in the first batch normalization module; the second series module includes a local space block (LS Block), a first layer normalization (LayerNorm) module and a two-dimensional selective scanning SS2D module connected in series in sequence; the third series module includes a second TSM module, a second layer normalization module and a residual gating block (RG Block) connected in series in sequence; wherein the output feature of the backbone network or the output feature of the pooling network is used as the input feature of the first series module, the output feature of the first series module is used as the input feature of the second series module, the output feature of the first series module is used as the residual branch to be fused with the output feature of the second series module, and the second fused feature obtained is used as the input feature of the third series module; the second fused feature is used as the residual branch to be fused with the output feature of the third series module, and the third fused feature obtained is used as the output feature of the SODSS module.

[0069] The SODSS module performs conventional convolution and batch normalization on the input features at the beginning, then enters the local space block and SS2D module, and then enhances the spatial features and temporal features through feature fusion and the second TSM module. Specifically, the input and output of the SODSS module can be:

[0070] 1. The input is features of three stages, and the size of each feature from top to bottom is, for example, (80, 80, 256), (40, 40, 256), and (20, 20, 256);

[0071] 2. First, input the feature into the first series module to obtain the a1 feature;

[0072] 3. Then the a1 feature is divided into two paths, the left path enters the next stage as the residual branch, and the right path passes through the second series module to obtain the a2 feature;

[0073] 4. The a1 feature is fused with the a2 feature to obtain the a3 feature;

[0074] 5. In the next step, the a3 feature is divided into two paths. The left path enters the next stage as a residual branch, and the right path passes through the third series module to obtain the a4 feature;

[0075] 6. Finally, feature a3 is fused with feature a4 to obtain output feature a5, where the feature sizes of a5 in the three stages can be (180, 180, 256), (80, 80, 256) and (40, 40, 256) respectively.

[0076] In one embodiment, optionally, Figure 5 As shown, the SS2D module includes a first linear change layer (Linear Layer) module, a first depth-separable convolution (DW-ConV) module, a second normalization module, and a second linear change layer module connected in series; wherein the first depth-separable convolution module includes a second activation function (Activation) module, and specifically, the second activation function module is embedded in the first depth-separable convolution module. The introduction of the input and output of SS2D can be referred to as follows:

[0077] 1. For a given input feature (Y1 feature), first enter the first linear change layer module to obtain the Y2 feature;

[0078] 2. The Y2 feature enters the first depth-separable convolution (DW-Conv module) module and the second activation function module to obtain the Y3 feature;

[0079] 3. The Y3 feature is input into the second normalization module and the second linear change layer module to obtain the Y4 feature.

[0080] In one embodiment, optionally, Figure 6 As shown, the local space block includes a second depth-separable convolution module, a second batch normalization module, a third convolution module, a third activation function module, and a fourth convolution module, which are sequentially connected in series; wherein the output features of the first series module are used as the input features of the second depth-separable convolution module, and the output features of the first series module are fused with the output features of the fourth convolution module, and the fourth fused features are obtained as the output features of the local space block. Figure 6 As shown in the figure, there is a residual branch in the local space block, and the attention mechanism is applied through the residual branch. The application of the attention mechanism can facilitate the capture of global features and improve the recognition accuracy.

[0081] The input and output description of the local space block can be specifically:

[0082] 1. For a given input feature (F1 feature), it first enters the second depth-separable convolution module, and then enters the second batch normalization module to obtain the F2 feature;

[0083] 2. The F2 feature enters the third convolution module, mixes the channel information and obtains the F3 feature through the third activation function module to better retain the information distribution;

[0084] 3. Finally, the F3 feature is fused with the F1 feature to obtain the output F4 feature.

[0085] In one embodiment, if Figure 7 As shown, the residual gating block includes a left branch sub-network, a right branch sub-network and a fifth convolution module; the left branch sub-network includes a fourth activation function module; the right branch sub-network includes a sixth convolution module and a fifth activation function module; wherein, as Figure 4 and Figure 7 As shown, the output features of the sixth convolution module are integrated with channel information and used as the input features of the fifth activation function module; the output features of the second-layer normalization module are used as the input features of the left branch sub-network and the right branch sub-network respectively; the features obtained by multiplying the output features of the left branch sub-network with the output features of the right branch sub-network are used as the input features of the fifth convolution module; the features obtained by adding the output features of the fifth convolution module with the output features of the second-layer normalization module are used as the output features of the residual gating block. In the left branch sub-network, the fourth activation function module uses the Gaussian error linear unit (GeLu) as the activation function to control the flow of information in each layer, merges with the right branch sub-network through element multiplication, and then performs a convolution operation through the fifth convolution module, so that the global features can be more refined, and the output features of the fifth convolution module are added to the original input features (the output features of the second-layer normalization module) to obtain the features as the output features of the residual gating block. Among them, the attention mechanism is applied in the residual gating block. The residual gating block can capture more global features through element multiplication, feature addition, and mixed channel information, but it only brings a slight increase in computational cost.

[0086] Among them, in the SODSS module, residual branches are used in many places, and the attention mechanism is applied. During the processing, feature fusion is performed in many places, and the fusion mechanism and mixed channel information are adopted, which is conducive to capturing global features; through the internal local space block, the local space information of the input feature can be effectively extracted to compensate for the lack of local modeling ability. The residual gating block can combine the idea of ​​gated aggregation with effective convolution and residual connection to effectively capture local dependencies and enhance the robustness of the model. By adopting the SODSS module, a series of processing is carried out at the input stage, so that the network can learn deeper and richer feature representations, and batch processing can keep the training and reasoning process efficient and stable.

[0087] It should be noted that the convolution module in the embodiment of the present application can be a two-dimensional convolution module, that is, a 1×1 convolution module.

[0088] Existing target detection technologies usually rely on Convolutional Neural Networks (CNN) or Regional Convolutional Neural Networks (R-CNN, Region-CNN) to identify vehicles, pedestrians and other targets in video images by training models. These methods have achieved remarkable results in various data sets, especially the single-task model has continuously improved the accuracy of target detection. However, these methods have the problems of high computational complexity and poor real-time performance, making them difficult to apply in resource-constrained vehicle systems.

[0089] Image quality recognition technology is mainly used to detect the clarity, brightness, contrast and other indicators of video images to ensure that drivers get the best visual experience. The existing methods mostly use technologies based on feature extraction and classifiers, such as using gradient, edge and other features combined with support vector machines (SVM) for classification. These methods work well when processing static images, but are not ideal for processing dynamic changes in video frame sequences, which can easily lead to misjudgment.

[0090] Image segmentation technology is used to segment different areas (such as roads, vehicles, pedestrians, etc.) in video images for further analysis and processing. Commonly used technologies include Fully Convolutional Neural Network (FCN), Image Segmentation Network (U-Net), etc., which can accurately segment the target area in the image. However, these methods usually require a lot of computing resources and training data, and have poor robustness when processing complex scenes.

[0091] The defects in the prior art are as follows:

[0092] The model has limited single-task processing capabilities: Most of the existing target detection technologies, image quality recognition technologies, and image segmentation technologies are single-task models, that is, each model can only handle one specific task. Although this approach can achieve good results on a single task, it requires multiple models to work together in multi-task scenarios, which increases the complexity and computational overhead of the system and reduces real-time performance and stability.

[0093] Lack of global modeling capabilities: Most existing technologies rely on the extraction and analysis of local features and lack the ability to model global information. This can easily lead to misjudgments and missed detections when dealing with complex scenes. For example, when driving at high speeds, rapidly changing scenes pose a challenge to local feature extraction.

[0094] Insufficient use of the temporal information of video images: The existing methods usually process each frame of the image independently, ignoring the correlation of the temporal information of the video image. This method performs well in static scenes, but in dynamic scenes, the lack of modeling of the relationship between consecutive frames leads to insufficient robustness of detection and recognition.

[0095] High computing resource requirements: In existing technologies, a large amount of computing resources and storage space are required during model training and inference, which places high demands on the hardware configuration of the vehicle system.

[0096] In the modern automobile industry, electronic rearview mirrors are a new driver assistance technology that is gradually replacing traditional optical rearview mirrors. The electronic rearview mirror system mainly uses a camera to collect rear road test videos and uses image processing technology to display clear rear images on the in-car display screen in real time. This system can not only eliminate the blind spots of traditional rearview mirrors, but also provide a wider field of view and improve driving safety.

[0097] However, the existing electronic rearview mirror system has many deficiencies in target detection, image quality recognition and image segmentation. The chip computing power used by the camera-monitor system (CMS) electronic rearview mirror system is only 5Tops, so the chip computing power cannot support the calculation and reasoning of multiple models at the same time. Therefore, in order to optimize this disadvantage and improve the algorithm recognition ability of the electronic rearview mirror system, this application provides a multi-task model, which is applied to the electronic rearview mirror system. The brain processor (Brain Processing Unit, BPU) of this model is only 1.5 times that of the conventional single-task model, which greatly reduces the delay of model processing.

[0098] The multi-task model provided in the embodiment of the present application can simultaneously realize the three tasks of target detection, image quality recognition, and image segmentation by designing a backbone network and three branch networks, simplifying the system architecture, reducing hardware resource requirements, and improving overall processing efficiency. The multi-task model can share the underlying feature extraction part through the backbone network, thereby realizing information complementarity and collaborative optimization among multiple tasks, improving the performance of each task, reducing repeated calculations, and improving the utilization of computing resources.

[0099] The technical solution provided by the embodiment of the present application, in terms of channels, by introducing the SODSS module, in the SODSS module, through multi-layer residuals, attention mechanisms, and fusion mechanisms, as well as mixing channel information during processing, rich global features can be extracted. In terms of timing, by introducing the TSM module, global features that reflect contextual information can be captured. Through the above two aspects, local modeling to global modeling can be achieved, and the overall information in each frame of video images can be better understood and analyzed, rather than relying solely on local features. This method enables the system to maintain high-precision target detection in complex scenarios, such as high-speed driving, bad weather and other conditions. At the same time, by introducing the timing information of the video image, the changes of the target in consecutive frames can be tracked, further improving the robustness of the detection and reducing misjudgments and missed detections.

[0100] The technical solution provided by the embodiment of the present application can enhance the image quality recognition capability, specifically: by introducing the above-mentioned SODSS module, TSM module, etc., combined with the timing information of the video image and the global modeling technology, the image quality can be evaluated more accurately. This not only includes static indicators such as clarity, brightness, contrast, etc., but also analyzes the impact of dynamic changes, thereby helping to adjust the display parameters in real time, ensuring that the driver can get the best visual experience in different environments, thereby improving driving safety.

[0101] The technical solution provided by this application can optimize the image segmentation results. Specifically, by using global modeling on channels and timing, and introducing the timing information of video images, it performs well in image segmentation tasks. Global modeling can capture the correlation between different regions in the image and enhance the accuracy and consistency of segmentation. The introduction of the timing information of video images can utilize the correlation between the previous and next frame video images, reduce the instability of the segmentation boundary, and improve the segmentation effect in dynamic scenes, which is particularly important for tasks such as road recognition and pedestrian detection.

[0102] The technical solution provided by this application can reduce the demand for computing resources. Specifically, by sharing the underlying feature extraction part through the backbone network, repeated calculations are reduced and the utilization rate of computing resources is improved. At the same time, combined with lightweight model design and efficient reasoning algorithms, the system can achieve real-time processing in a resource-limited vehicle environment. This not only reduces hardware costs, but also improves the response speed and stability of the system.

[0103] The technical solution provided in the embodiment of the present application can improve the scalability and adaptability of the system. Specifically: the multi-task model provided in the present application has good scalability, and can flexibly adjust and expand different task modules according to specific application requirements. For example, branches can be added, and functional modules such as driver fatigue detection and road sign recognition can be added on the original basis, or driver fatigue detection and road sign recognition training can be added during training, so that the model can increase driver fatigue detection and road sign recognition, and further improve the intelligence level of the system. In addition, the model has strong adaptability and can operate stably in different vehicle models and complex driving environments to meet diverse market needs.

[0104] The technical solution provided by the embodiment of the present application can improve user experience and driving safety. Specifically, through the multi-task model provided by the present application, target detection, image quality recognition and image segmentation can be performed efficiently, which can significantly improve the overall performance of the electronic rearview mirror system, provide clearer, more stable and reliable rear view information, not only enhance the user's driving experience, but also greatly improve driving safety; it can enable the driver to discover potential dangers earlier, make timely responses, and reduce the occurrence of traffic accidents.

[0105] Figure 8 1 is a flowchart of a processing method based on an image processing multi-task model provided in an embodiment of the present application. The method is applied to an electronic rearview mirror system and to vehicle autonomous driving perception. Figure 8 As shown, the method comprises the following steps:

[0106] Step 801: Acquire continuous video images of a drive test video.

[0107] Step 802: Input the continuous video image into an image processing multi-task model to obtain a first target detection result, a second target detection result, a third target detection result, a lane line detection result, and an image quality classification result; wherein the image processing multi-task model is the image processing multi-task model provided in an embodiment of the present application, and the model is a trained model.

[0108] In one embodiment, the method provided by the present application may further include: determining an evaluation score for each video image based on the image quality classification result, and determining the quality of a batch of video images based on the evaluation score of each video image. Specifically, continuous video images may be input into the multi-task model, for example, 8 continuous video images may be input into the multi-task model to obtain image quality classification results. The image quality classification results may include clear, general, fuzzy and meaningless, and background text. The reference standard for classification may refer to Fig. 9. The evaluation score of each video image can be determined according to the image quality classification, wherein the evaluation scores corresponding to clear, general, blurry and meaningless, and background text can be 4 points, 3 points, 2 points, and 1 point, respectively. The evaluation score of each batch of video images can be processed into a score of 0-1, that is, normalized; so the evaluation score of each batch of video images can be score = (4×clear+3×general+2×blurry+1×background) / 4×total; the quality of each batch of video images is determined by the evaluation score of each batch of video images; wherein, the higher the quality, the higher the evaluation score, and whether to process the video images in the future is determined according to the quality of the video images.

[0109] Therefore, by determining the quality of each batch of video images, the quality of the video images can be understood, which facilitates the subsequent processing of each batch of video images.

[0110] Fig.10 is a flow chart of a training method for an image processing multi-task model provided in an embodiment of the present application, such as Fig.10 As shown, the method comprises the following steps:

[0111] Step 901: Obtain training samples of the image processing multi-task model, and annotate continuous video images in the training samples.

[0112] Specifically, this step is applied before the processing method based on the image processing multi-task model is used, and the image processing multi-task model can be trained before the image processing multi-task model is used.

[0113] A specific training method may be to obtain training samples of the image processing multi-task model and annotate continuous video images in the training samples, wherein the specific annotations may include quality classification of the video image, target position in the video image, target classification, lane lines and other information.

[0114] Step 902: Use labeled continuous video images to train the image processing multi-task model to obtain a trained image processing multi-task model; wherein the image processing multi-task model is the image processing multi-task model provided in an embodiment of the present application.

[0115] Specifically, the labeled continuous video images are input into the image processing multi-task model, the output results are compared with the standard results, and the image processing multi-task model is optimized based on the loss function to obtain the trained image processing multi-task model. Among them, the loss function of target detection can be a cross entropy loss function, a CIOU loss function, and a BCE loss function, the loss function of lane line detection can be a pixel-by-pixel cross entropy loss function; the loss function of image quality classification can be a cross entropy loss function.

[0116] Fig.11is a structural block diagram of a processing device based on an image processing multi-task model provided in an embodiment of the present application, such as Fig.11 As shown, including:

[0117] An acquisition module 101 is used to acquire continuous video images of a drive test video;

[0118] The processing module 102 is used to input the continuous video image into the image processing multi-task model to obtain a first target detection result, a second target detection result, a third target detection result, a lane line detection result and an image quality classification result.

[0119] Fig.12 is a structural block diagram of a training device for an image processing multi-task model provided in an embodiment of the present application, such as Fig.12 As shown, the device comprises:

[0120] A training sample determination unit 201 is used to obtain training samples of the image processing multi-task model and annotate continuous video images in the training samples;

[0121] The training unit 202 is used to train the image processing multi-task model using annotated continuous video images to obtain a trained image processing multi-task model; wherein the image processing multi-task model is the image processing multi-task model provided in the embodiment of the present application.

[0122] like Fig.13 As shown, an embodiment of the present application provides an electronic device, including a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0123] Memory 113, used for storing computer programs;

[0124] In one embodiment of the present application, the processor 111 is used to implement the processing method based on the image processing multi-task model or the training method of the image processing multi-task model provided by any of the aforementioned method embodiments when executing the program stored in the memory 113.

[0125] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the processing method based on a multi-task model or the training method of a multi-task model provided in any of the aforementioned method embodiments are implemented.

[0126] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0127] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0128] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0129] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A multi-task model for image processing, characterized in that: include: A backbone network, a pooled network, a first branch network, a second branch network, and a third branch network; The backbone network is used to perform batch processing on continuous video images in the drive test video, and to add time sequence information of the video images during the batch processing of the continuous video images to obtain output features; The first branch network includes three parallel sub-networks, namely a first sub-network, a second sub-network and a third sub-network; The output features of the backbone network are processed by the pooling network and the first sub-network to obtain a first target detection feature; The output features of the backbone network are respectively input into the second sub-network and the third sub-network to obtain corresponding second target detection features and third target detection features; The sub-network head includes an optimized deep semantic perception module; the optimized deep semantic perception module is used to obtain deep semantic features reflecting target features through multiple different types of attention mechanisms and fusion mechanisms; The second branch network is used to fuse the deep semantic features output by each of the optimized deep semantic perception modules to obtain a first fused feature, and process the first fused feature accordingly to obtain a lane segmentation result; The output features of the backbone network are processed by the pooling network and the third branch network to obtain an image quality classification result.

2. The model according to claim 1, characterized in that The backbone network includes a first convolution module and a target module formed by a plurality of target series layers in parallel; wherein the first convolution module and the target module are connected in series; the target series layer is composed of a first convolution normalization and activation CBS module and a first space and spatiotemporal fusion C3T module connected in series; The C3T module includes an upper branch sub-network, a lower branch sub-network and a second CBS module; The upper branch sub-network includes a third CBS module and a bottleneck module; The lower branch sub-network includes a fourth CBS module and a first time shift TSM module; Among them, the output features of the upper sub-branch network and the output features of the lower sub-branch network are fused as the input features of the second CBS module.

3. The model according to claim 1, characterized in that The optimized deep semantic perception module includes: a first series module, a second series module and a third series module; The first series module includes a second convolution module and a first batch of normalization modules connected in series in sequence; wherein the first batch of normalization modules includes a first activation function module; The second series module includes a local space block, a first layer normalization module and a two-dimensional selective scanning SS2D module connected in series in sequence; The third series module includes a second TSM module, a second layer normalization module and a residual gating block connected in series in sequence; Among them, the output features of the backbone network or the output features of the pooling network are used as input features of the first series module, the output features of the first series module are used as input features of the second series module, the output features of the first series module are used as residual branches to be fused with the output features of the second series module, and the second fused features are used as input features of the third series module; the second fused features are used as residual branches to be fused with the output features of the third series module, and the third fused features are used as output features of the optimized deep semantic perception module.

4. The model according to claim 3, characterized in that The SS2D module includes a first linear change layer module, a first depth-separable convolution module, a second normalization module and a second linear change layer module which are connected in series in sequence; wherein the first depth-separable convolution module includes a second activation function module.

5. The model according to claim 3, characterized in that The local space block includes a second depth-separable convolution module, a second batch normalization module, a third convolution module, a third activation function module and a fourth convolution module connected in series in sequence; Among them, the output features of the first series module are used as input features of the second depth-wise separable convolution module, the output features of the first series module are fused with the output features of the fourth convolution module, and the obtained fourth fused features are used as output features of the local spatial block.

6. The model according to claim 3, characterized in that The residual gating block includes a left branch sub-network, a right branch sub-network and a fifth convolution module; The left branch sub-network includes a fourth activation function module; The right branch sub-network includes a sixth convolution module and a fifth activation function module; wherein the output feature of the sixth convolution module is fused with channel information as the input feature of the fifth activation function module; The output features of the second-layer normalization module are used as input features of the left branch sub-network and the right branch sub-network respectively; the features obtained by multiplying the output features of the left branch sub-network and the output features of the right branch sub-network are used as input features of the fifth convolution module; the features obtained by adding the output features of the fifth convolution module and the output features of the second-layer normalization module are used as output features of the residual gating block.

7. The model according to claim 1, characterized in that The first sub-network also includes a seventh convolution module, which is used to perform a convolution operation on the deep semantic features output by the corresponding optimized deep semantic perception module to obtain a first target detection feature; The second sub-network also includes an eighth convolution module, which is used to perform a convolution operation on the deep semantic features output by the corresponding optimized deep semantic perception module to obtain a second target detection feature; The third sub-network also includes a ninth convolution module, which is used to perform a convolution operation on the deep semantic features output by the corresponding optimized deep semantic perception module to obtain a third target detection feature; The second branch network includes a fusion module, a first upsampling module, a tenth convolution module and a second upsampling module connected in series in sequence; wherein the deep semantic features output by each of the optimized deep semantic modules are used as input features of the fusion module, and the output result of the second upsampling module is used as the lane line segmentation result; The third branch network includes a second C3T module, an eleventh convolutional module and a fully connected module connected in series in sequence; The output features of the pooling network are used as input features of the second C3T module, and the output result of the fully connected module is used as the image quality classification result.

8. A processing method based on an image processing multi-task model, characterized in that: include: Acquire continuous video images of the drive test video; The continuous video images are input into the image processing multi-task model described in any one of claims 1-7 to obtain a first target detection result, a second target detection result, a third target detection result, a lane line detection result and an image quality classification result.

9. The method according to claim 8, characterized in that Also includes: An evaluation score for each video image is determined based on the image quality classification result, and the quality of the batch of video images is determined based on the evaluation score for each video image.

10. A training method for a multi-task model for image processing, characterized in that: include: Obtaining training samples of the image processing multi-task model, and annotating continuous video images in the training samples; The image processing multi-task model is trained using labeled continuous video images to obtain a trained image processing multi-task model; wherein the image processing multi-task model is the image processing multi-task model described in any one of claims 1-7.

Images