Method and system for establishing automatic driving image recognition model

Abstract

The invention provides a method and system for establishing an automatic driving image recognition model, and the method comprises the steps: obtaining original video images with continuous time in a vehicle driving process, and enabling the original video images to serve as a set; dividing each original video image in the set into sub-blocks of a preset scale, performing priority ranking on the sub-blocks, determining to-be-processed areas corresponding to the original video images, and performing data processing on the to-be-processed areas in the original video images in sequence to obtain enhanced images with continuous time as training data; and marking an obstacle on the enhanced image in the training data as a training label, training a preset recognition model by using the training data and the training label, and obtaining an automatic driving image recognition model after the training is finished. According to the method, data processing is only performed on the sub-blocks with higher processing priority ranking according to processing requirements, so that the workload and the calculation complexity of an image processor and a visual perception processor of the automatic driving system are reduced.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method and system for establishing an autonomous driving image recognition model. Background Art

[0002] In the field of autonomous driving, vehicles can collect information about their surroundings and status as a basis for subsequent decision-making.

[0003] In the related technology, video images are often collected through on-board video surveillance. However, when an autonomous driving vehicle is driving, the random vibration and bumping of the vehicle will cause the collected video images to have visual random jitters, and the video images will not be clear and contain a lot of noise, which will affect the control of the autonomous driving and cause poor vehicle control accuracy. In addition, image processing is usually performed on the entire video image to solve the above problems, which increases the additional workload and computational complexity of the background processor.

[0004] Based on the above analysis of the development status of this technical field, the existing technical solutions lack an autonomous driving graphic recognition model that is applied in the field of autonomous driving and only processes graphic blocks with higher importance. Summary of the invention

[0005] The purpose of the present invention is to provide a method and system for establishing an autonomous driving image recognition model, aiming to solve the above-mentioned problems in the prior art.

[0006] According to a first aspect of an embodiment of the present invention, a method for establishing an autonomous driving image recognition model is provided, comprising:

[0007] Acquire time-continuous raw video images during vehicle driving as a set;

[0008] Divide each original video image in the set into sub-blocks of a preset size, determine the to-be-processed area corresponding to each original video image by sorting the sub-blocks by priority, perform data processing on the to-be-processed areas in the original video images in turn, and obtain a time-continuous enhanced image as training data;

[0009] The obstacles on the enhanced images in the training data are marked as training labels, and the preset recognition model is trained using the training data and training labels. After the training, an autonomous driving image recognition model is obtained.

[0010] According to a second aspect of an embodiment of the present invention, a system for establishing an autonomous driving image recognition model is provided, comprising:

[0011] A data acquisition module, used for acquiring time-continuous original video images during vehicle driving as a set;

[0012] A processing module is used to divide each original video image in the set into sub-blocks of a preset size, determine the to-be-processed area corresponding to each original video image by prioritizing the sub-blocks, and sequentially process the to-be-processed areas in the original video images to obtain a time-continuous enhanced image as training data;

[0013] The training module is used to mark obstacles on the enhanced image in the training data as training labels, and use the training data and training labels to train the preset recognition model. After the training, the autonomous driving image recognition model is obtained.

[0014] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of the method for establishing an autonomous driving image recognition model as provided in the first aspect of the present disclosure are implemented.

[0015] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which a program for implementing information transmission is stored. When the program is executed by a processor, the steps of the method for establishing an autonomous driving image recognition model provided in the first aspect of the present disclosure are implemented.

[0016] The technical solution provided by the embodiment of the present invention includes the following beneficial effects: each frame of the original video image is divided into multiple sub-blocks, and only the sub-blocks with higher processing priority are processed according to the processing requirements, while the sub-blocks with lower importance are not processed, thereby reducing the workload and computational complexity of the image processor and visual perception processor of the autonomous driving system.

[0017] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0019] Figure 1 is a flowchart of a method for establishing an autonomous driving image recognition model according to an embodiment of the present invention;

[0020] Figure 2 It is a schematic diagram of model establishment and application of an embodiment of the present invention;

[0021] Figure 3is a schematic diagram of a system for establishing an autonomous driving image recognition model according to an embodiment of the present invention;

[0022] Figure 4 is a schematic diagram of system module integration of an embodiment of the present invention;

[0023] Figure 5 is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the following will be combined with the drawings in one or more embodiments of this specification to clearly and completely describe the technical solutions in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.

[0025] Method Embodiment

[0026] According to an embodiment of the present invention, a method for establishing an automatic driving image recognition model is provided. Figure 1 is a flow chart of a method for establishing an automatic driving image recognition model according to an embodiment of the present invention. Figure 1 As shown, the method for establishing an autonomous driving image recognition model according to an embodiment of the present invention specifically includes:

[0027] In step S110, obtaining temporally continuous original video images during the vehicle driving process as a set includes:

[0028] The vehicle driving monitoring data of the autonomous driving vehicle is obtained, and the on-board video is extracted from the vehicle driving monitoring data, and each frame image in the video is an original video image.

[0029] In step S120, each original video image in the set is divided into sub-blocks of a preset size, and the to-be-processed area corresponding to each original video image is determined by prioritizing the sub-blocks, and the to-be-processed area in the original video image is processed in turn to obtain a time-continuous enhanced image as training data, which specifically includes:

[0030] The sub-blocks in the original video image are divided into grids in the horizontal and vertical directions. In addition to the traditional grid division in both the horizontal and vertical directions, preferably, only the horizontal division or only the vertical division can be performed.

[0031] The priority order is determined according to the specific requirements in the application. In the embodiment of the present invention, the requirement is to reduce jitter and blur. Preferably, if there are many specific requirements that cannot be clearly defined, a machine learning recognition algorithm can be used to obtain the correlation between the current image sub-block and the image visual perception requirement, and a targeted data processing method can be used based on the correlation.

[0032] In an embodiment of the present invention, the data processing method is to use the average gradient method to calculate the blur degree of each sub-block, sort the sub-blocks in the original video image from high to low according to the blur degree, and select a threshold number of sub-blocks in the priority sorting as the to-be-processed area corresponding to the original video image. In addition, preferably, the priority sorting can also be adjusted by engineering personnel according to actual conditions, or a prediction algorithm can be used to detect the importance of the image sub-block. The prediction algorithm detection can be determined by the detection results of continuous frames. In an embodiment of the present invention, a deep learning model VGGNet is used to extract the features of the sub-blocks, calculate the importance scores of the features, and perform priority sorting based on the importance scores. The VGGNet model uses multiple small convolution kernel stacks, which is suitable for local feature extraction.

[0033] That is, only the image sub-blocks with higher importance are processed, while the image sub-blocks with lower importance are not processed, which greatly reduces the workload and computational complexity of the image processor and visual perception processor in the autonomous driving system.

[0034] The specific process of data processing includes:

[0035] Preferably, all sub-blocks in each original video image may be pre-processed before data processing, which may include: denoising, demosaicing, sharpening, ring point removal, black level removal, automatic exposure, automatic focus, automatic white balance, high dynamic range processing, gamma mapping, color enhancement, color space conversion, image distortion correction, etc.;

[0036] Obtain the next frame of the original video image that is time-continuous with the current original video image, and determine the position of the current area to be processed in the next frame of the original video image through the sub-block matching algorithm. It should be noted that the area to be processed in each original video image is not the same, and each frame of the original video image needs to be processed in sequence;

[0037] However, since each frame of the original video image is divided in the same preset scale, the sub-block matching algorithm can correspond to the corresponding position of each sub-block in the current area to be processed in the next frame of the original video image.

[0038] Calculate the global motion vector between the current processing area of ​​the two frames of original video images, calculate the motion compensation vector according to the global motion vector, use the motion compensation vector to perform inverse correction on the processing area of ​​the current original video image, and obtain the corresponding processed jittered image; perform data enhancement and sparse compression on the processing area in the processed jittered image to obtain an enhanced image, wherein sparse compression refers to an image compression algorithm based on sparse representation, which can improve video quality and resolution compared with traditional video transmission, and is suitable for video image transmission of vehicle-mounted video monitoring systems, so that subsequent judgments on the current driving action to be performed are more accurate;

[0039] The original video images are processed in sequence to obtain the corresponding enhanced images.

[0040] In step S130, obstacles on the enhanced image in the training data are marked as training labels, and the preset recognition model is trained using the training data and the training labels. After the training, an autonomous driving image recognition model is obtained, which specifically includes:

[0041] The training data and training labels can be used as a training sample set, and the sample set is transmitted to a database for storage. The number of obstacles that can be marked in the enhanced image is greater than the number of obstacles that can be marked in the corresponding initial video image. The data processing method of the embodiment of the present invention can improve image quality.

[0042] The method further comprises:

[0043] In step S140, the autonomous driving image recognition model is used to identify obstacles around the vehicle, the driving action is determined based on avoiding the obstacles, and the operation instruction is received to execute the driving action. Specifically, it includes:

[0044] In actual applications, the real-time surrounding scenes of the vehicle are input into the autonomous driving image recognition model, and the vehicle can be directed to operate according to the recognition results, and the driving action that needs to be performed at present can be determined. The operation instructions are detailed operation instructions for the corresponding driving actions;

[0045] Receiving a first operation instruction autonomously sent by the vehicle or a second operation instruction sent by the main control console to perform a driving action, wherein the first operation instruction includes a steering wheel action, an accelerator action and a brake action, and the second operation instruction includes a start action and a stop action;

[0046] The throttle action and brake action are used to control the acceleration or deceleration of the speed respectively. They are routine operations during vehicle driving and can be decided independently according to environmental changes. The start action and stop action are used to start and stop the vehicle respectively. They are crucial to traffic safety and can coordinate the management of the start and stop time between multiple vehicles to ensure smooth traffic.

[0047] Figure 2Schematic diagram of model establishment and application of an embodiment of the present invention, such as Figure 2 As shown in the figure, the basic process of model establishment is demonstrated. The established autonomous driving image recognition model can be used for recognition and judgment in practical applications.

[0048] To sum up, in response to the existing problems, the method for establishing an autonomous driving image recognition model invented in this invention divides each frame of original video image into multiple sub-blocks, and only processes data for sub-blocks with higher processing priority according to processing requirements, while sub-blocks with lower importance are not processed, thereby reducing the workload and computational complexity of the image processor and visual perception processor of the autonomous driving system; the to-be-processed area of ​​the original video image is reversely corrected and data enhanced in turn, so that the number of obstacles that can be marked in the enhanced image is higher than that in the original video image, thereby improving the quality and resolution of the video image; the established autonomous driving image recognition model can be used in actual autonomous driving applications, can assist in determining the driving action that needs to be performed currently, and receive operation instructions sent by different subjects according to the driving action, thereby improving the safety and availability of the vehicle during driving.

[0049] System Example

[0050] According to an embodiment of the present invention, a system for establishing an autonomous driving image recognition model is provided. Figure 3 is a schematic diagram of a system for establishing an automatic driving image recognition model according to an embodiment of the present invention. Figure 3 As shown, the system for establishing an autonomous driving image recognition model according to an embodiment of the present invention specifically includes:

[0051] The data acquisition module 30 is used to obtain time-continuous original video images during the driving process of the vehicle as a set.

[0052] The processing module 32 is used to divide each original video image in the set into sub-blocks of a preset size, determine the to-be-processed area corresponding to each original video image by prioritizing the sub-blocks, and sequentially process the to-be-processed areas in the original video images to obtain a time-continuous enhanced image as training data, specifically for:

[0053] The sub-blocks in the original video image are divided into a grid in horizontal and vertical directions.

[0054] The average gradient method is used to calculate the blur degree of each sub-block, and the sub-blocks in the original video image are sorted from high to low according to the blur degree. A threshold number of sub-blocks in the priority sorting are selected as the corresponding original video image The area to be processed.

[0055] Obtain the next frame of the original video image that is time-continuous with the current original video image, determine the position of the current area to be processed in the next frame of the original video image by using a sub-block matching algorithm, calculate the global motion vector between the current area to be processed of the two frames of the original video image, calculate the motion compensation vector according to the global motion vector, and use the motion compensation vector to perform inverse correction on the area to be processed of the current original video image to obtain the corresponding processed jittered image; perform data enhancement and sparse compression on the area to be processed in the processed jittered image to obtain an enhanced image;

[0056] The original video images are processed in sequence to obtain the corresponding enhanced images.

[0057] The training module 34 is used to mark obstacles on the enhanced image in the training data as training labels, and use the training data and training labels to train a preset recognition model. After the training, an autonomous driving image recognition model is obtained.

[0058] The device further comprises:

[0059] The application module 36 is used to use the automatic driving image recognition model to identify obstacles around the vehicle, determine the driving action based on avoiding obstacles, and receive the operation instruction to perform the driving action. The first operation instruction sent autonomously by the vehicle or the second operation instruction sent by the main control console is received to perform the driving action, wherein the first operation instruction includes the steering wheel action, the throttle action and the brake action, and the second operation instruction includes the start action and the stop action.

[0060] Figure 4 Schematic diagram of system module integration of an embodiment of the present invention, such as Figure 4 As shown, the module architecture of integrating model training and application functions into one system is demonstrated. The recognition module is used to identify obstacles based on processed verification data, the judgment module is used to judge the driving action that needs to be performed currently, and the execution module is used to receive instructions to perform operations corresponding to the driving action. It is divided into a first execution module and a second execution module.

[0061] To sum up, in response to the existing problems, the system for establishing an autonomous driving image recognition model invented in this invention divides each frame of original video image into multiple sub-blocks, and only processes data for sub-blocks with higher processing priority according to processing requirements, while sub-blocks with lower importance are not processed, thereby reducing the workload and computational complexity of the image processor and visual perception processor of the autonomous driving system; the to-be-processed area of ​​the original video image is reversely corrected and data enhanced in turn, so that the number of obstacles that can be marked in the enhanced image is higher than that in the original video image, thereby improving the quality and resolution of the video image; the established autonomous driving image recognition model can be used in actual autonomous driving applications, can assist in determining the driving action that needs to be performed currently, and receive operation instructions sent by different subjects according to the driving action, thereby improving the safety and availability of the vehicle during driving.

[0062] Electronic device embodiment

[0063] Figure 5 Schematic diagram of an electronic device according to an embodiment of the present invention. The electronic device 500 may include at least one processor 510 and a memory 520. The processor 510 may execute instructions stored in the memory 520. The processor 510 is connected to the memory 520 through a data bus. In addition to the memory 520, the processor 510 may also be connected to an input device 530, an output device 540, and a communication device 550 through a data bus.

[0064] The processor 510 may be any conventional processor, such as a commercially available CPU. The processor may also include a graphics processor (Graphic Process Unit, GPU), a field programmable gate array (Field Programmable Gate Array, FPGA), a system on chip (System on Chip, SOC), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC) or a combination thereof.

[0065] The memory 520 may be implemented by any type of volatile or nonvolatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0066] In the embodiment of the present disclosure, executable instructions are stored in the memory 520, and the processor 510 can read the executable instructions from the memory 520 and execute the instructions to implement all or part of the steps of the method for establishing an autonomous driving image recognition model in any of the above-mentioned exemplary embodiments.

[0067] Computer Readable Storage Medium Embodiments

[0068] In addition to the above-mentioned methods and systems, exemplary embodiments of the present disclosure may also be a computer program product or a computer-readable storage medium storing the computer program product. The computer product includes computer program instructions, which can be executed by a processor to implement all or part of the steps described in the method for establishing an autonomous driving image recognition model in any of the above-mentioned exemplary embodiments.

[0069] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages ​​and scripting languages ​​(e.g., Python). The program code may be executed entirely on the user computing device, partially on the user computing device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0070] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of readable storage media include: a static random access memory (SRAM) with one or more wires electrically connected, an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk or an optical disk, or any suitable combination of the above.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for establishing an automatic driving image recognition model, characterized in that: include: Acquire time-continuous original video images during vehicle driving as a set; Divide each original video image in the set into sub-blocks of a preset size, determine the to-be-processed area corresponding to each original video image by prioritizing the sub-blocks, and sequentially process the to-be-processed areas in the original video images to obtain a time-continuous enhanced image as training data; The obstacles on the enhanced image in the training data are marked as training labels, and the preset recognition model is trained using the training data and the training labels. After the training, an autonomous driving image recognition model is obtained.

2. The method according to claim 1, characterized in that The method further comprises: The autonomous driving image recognition model is used to identify obstacles around the vehicle, driving actions are determined based on avoiding the obstacles, and operation instructions are received to execute the driving actions.

3. The method according to claim 1, characterized in that The step of dividing each original video image in the set into sub-blocks of a preset size specifically includes: dividing the sub-blocks in the original video image in a grid form in horizontal and vertical directions.

4. The method according to claim 1, characterized in that: The step of determining the to-be-processed area corresponding to each original video image by prioritizing the sub-blocks specifically includes: The blur degree of each sub-block is calculated using the average gradient method, and the sub-blocks in the original video image are prioritized according to the blur degree from high to low, and a threshold number of sub-blocks in the priority ranking are selected as the to-be-processed area corresponding to the original video image.

5. The method according to claim 1, characterized in that The step of sequentially processing the to-be-processed regions in the original video image to obtain a temporally continuous enhanced image as training data specifically includes: Acquire the next frame of original video image that is time-continuous with the current original video image, determine the position of the current area to be processed in the next frame of original video image by using a sub-block matching algorithm, calculate the global motion vector between the current area to be processed of the two frames of original video images, calculate the motion compensation vector according to the global motion vector, and use the motion compensation vector to perform inverse correction on the area to be processed of the current original video image to obtain a corresponding processed jittered image; perform data enhancement and sparse compression on the area to be processed in the processed jittered image to obtain an enhanced image; The original video images are processed in sequence to obtain corresponding enhanced images.

6. The method according to claim 2, characterized in that The receiving the operation instruction to execute the driving action specifically includes: Receive a first operating instruction autonomously sent by the vehicle or a second operating instruction sent by the main control console to perform the driving action, wherein the first operating instruction includes a steering wheel action, an accelerator action and a brake action, and the second operating instruction includes a start action and a stop action.

7. A system for establishing an autonomous driving image recognition model, characterized in that: include: A data acquisition module, used for acquiring time-continuous raw video images during vehicle driving as a set; A processing module, used for dividing each original video image in the set into sub-blocks of a preset size, determining the to-be-processed area corresponding to each original video image by prioritizing the sub-blocks, and sequentially performing data processing on the to-be-processed areas in the original video images to obtain a time-continuous enhanced image as training data; The training module is used to mark obstacles on the enhanced image in the training data as training labels, and use the training data and the training labels to train a preset recognition model. After the training, an autonomous driving image recognition model is obtained.

8. The system according to claim 7, characterized in that The system further comprises: The application module is used to use the autonomous driving image recognition model to identify obstacles around the vehicle, determine driving actions based on avoiding the obstacles, and receive operation instructions to execute the driving actions.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method for establishing an autonomous driving image recognition model as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an implementation program for information transmission, and when the program is executed by the processor, the steps of the method for establishing an autonomous driving image recognition model as described in any one of claims 1 to 6 are implemented.

Images