Camera image processing method and device and electronic equipment

Abstract

The invention provides a camera image processing method and device and electronic equipment, and relates to the field of image processing, and the method comprises the steps: determining a distortion parameter corresponding to a camera parameter from a preset database according to the camera parameter corresponding to a current image under the condition of determining that the computing power of a current CPU module is insufficient; sending the current image and the distortion parameter to a GPU module, indicating the GPU module to create a VPI flow according to the current image and the distortion parameter, and processing the current image in the VPI flow according to the distortion parameter to obtain a processed image; and receiving the processed image until all the images are processed. According to the method, the opportunity of executing the image processing task by the GPU is dynamically determined through various detection strategies, the computing power of the CPU is saved, the stability of the system under various load conditions is ensured, and when the computing power of the CPU is insufficient, the advantages of the GPU in the aspect of image processing are fully exerted by repeatedly executing the step of processing the image on the GPU module, so that the image processing efficiency and speed are improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a camera image processing method, device and electronic equipment. Background Art

[0002] The rapid development of autonomous driving technology has brought about a strong demand for efficient, fast, and stable image processing. In autonomous driving scenarios, cameras play a vital role in implementing key algorithms such as obstacle detection and lane line detection. However, due to the huge amount of image processing data, traditional CPU computing power can no longer meet the real-time, high frame rate processing requirements, resulting in performance bottlenecks when processing images in complex scenarios.

[0003] In embedded scenarios, especially in autonomous driving systems, limited CPU computing power is one of the key factors that restrict the overall performance of the system. Traditional image processing methods cannot meet the requirements for a stable 30-frame image frame rate, which is essential for real-time perception and decision-making. Therefore, it is urgent to find a solution that can improve image processing efficiency, increase frame rate and maintain stability. With the advantages of parallel processing of the Graphics Processing Unit (GPU), heavy image processing tasks can be accelerated, thereby effectively improving the image processing speed and achieving fast and stable image processing. However, this will lead to increased data transmission and communication costs, resulting in performance loss. Using the CPU alone for image processing has insufficient computing power. How to reasonably allocate tasks to the CPU and GPU, make full use of the advantages of both, and improve the speed and efficiency of image processing while ensuring computing performance has become a technical problem that needs to be solved urgently.

[0004] However, there is currently no technical solution that can solve the above technical problems, and there is no camera image processing method, device and electronic equipment. Summary of the invention

[0005] The present invention provides a camera image processing method, device and electronic device. By reasonably allocating tasks to the CPU and GPU, the advantages of both can be fully utilized to improve the speed and efficiency of image processing while ensuring computing performance. It is necessary to select appropriate computing resources for processing according to actual conditions to achieve the best image processing effect.

[0006] In a first aspect, the present invention provides a camera image processing method, comprising:

[0007] For any image among all the images, if it is determined that the current CPU module has insufficient computing power, repeat the following steps:

[0008] According to the camera parameters corresponding to the current image, determining the distortion parameters corresponding to the camera parameters from a preset database;

[0009] Sending the current image and the distortion parameters to a GPU module, instructing the GPU module to create a computer vision programming interface (Vision Programming Interface, VPI) stream according to the current image and the distortion parameters, and processing the current image according to the distortion parameters in the VPI stream to obtain a processed image;

[0010] Receiving processed images sent from the GPU module until all images are processed;

[0011] The preset database includes a correspondence between each preset camera parameter and a preset distortion parameter corresponding to the preset camera parameter.

[0012] According to the camera image processing method provided by the present invention, for any image among all images, when it is determined that the computing power of the current CPU module is sufficient, the steps further include:

[0013] Inputting the current image into a preset image processing model to obtain a processed image output by the preset image processing model;

[0014] The preset image processing model is determined after training based on all sample images carrying distortion parameter labels and processed sample images corresponding to each sample image.

[0015] According to the camera image processing method provided by the present invention, before determining the distortion parameters corresponding to the camera parameters from a preset database according to the camera parameters corresponding to the current image, the method further includes:

[0016] For any processing task, if it is determined that the time for the current CPU module to execute the processing task exceeds the preset processing time, it is determined that the computing power of the current CPU module is insufficient;

[0017] Or, using a system monitoring tool to obtain the CPU module utilization, and when it is determined that the current CPU module utilization is greater than a preset utilization, determining that the current CPU module computing power is insufficient;

[0018] For any response operation, when it is determined that the response time is longer than the preset response time, it is determined that the current CPU module computing power is insufficient;

[0019] If an error message, freeze message, or crash occurs, determine that the current CPU module has insufficient computing power.

[0020] According to the camera image processing method provided by the present invention, the distortion parameters include radial distortion parameters and tangential distortion parameters, and the current image is processed according to the distortion parameters in the VPI stream to obtain a processed image, including:

[0021] Processing the current image according to the radial distortion parameter and the tangential distortion parameter to obtain a dedistorted image;

[0022] The dedistorted image is cropped and scaled to obtain the processed image.

[0023] According to the camera image processing method provided by the present invention, the current image is processed according to the radial distortion parameter and the tangential distortion parameter to obtain a dedistorted image, including:

[0024] x′=x(1+k1r 2 +k2r 4 +k3r 6 )+2p1xy+p2(r 2 +2x 2 )

[0025] y′=y(1+k1r 2 +k2r 4 +k3r 6 )+2p2xy+p1(r 2 +2y 2 )

[0026] Where x is the X-axis coordinate of the current image, y is the Y-axis coordinate of the current image, x′ is the X-axis coordinate of the dedistorted image, y′ is the Y-axis coordinate of the dedistorted image, k1, k2, k3 are radial distortion parameters, p1, p2 are tangential distortion parameters, r is the algebraic value,

[0027] According to the camera image processing method provided by the present invention, the dedistorted image is cropped and scaled to obtain the processed image, including:

[0028] Instructing the GPU module to call a preset configuration parameter NvBufferRect to crop the dedistorted image to obtain a cropped image;

[0029] The GPU module is instructed to call the NvBufsurfTransform function provided by Nvidia to scale the cropped image to obtain the processed image.

[0030] According to the camera image processing method provided by the present invention, after processing the current image according to the distortion parameter in the VPI stream to obtain the processed image, the method further includes:

[0031] Instruct the GPU module to call the JPEG codec library NvJpeg to compress the processed image and specify the quality score of the compressed image;

[0032] Instruct the GPU module to send the compressed image to the current CPU module.

[0033] According to the camera image processing method provided by the present invention, before sending the current image and the distortion parameters to the GPU module, the method further includes:

[0034] For any image among all the images, convert the original color format of the image into YUV420 format;

[0035] All the images are determined in an autonomous driving scenario based on the video stream read in by the underlying driver.

[0036] In a second aspect, a camera image processing device is provided, comprising a repeating unit, wherein the repeating unit comprises a determining unit, an indicating unit, and a receiving unit;

[0037] For any image among all the images, when it is determined that the computing power of the current CPU module is insufficient, the determining unit is used to determine the distortion parameters corresponding to the camera parameters from a preset database according to the camera parameters corresponding to the current image;

[0038] The instructing unit is used to send the current image and the distortion parameters to the GPU module, instruct the GPU module to create a computer vision programming interface VPI stream according to the current image and the distortion parameters, and process the current image according to the distortion parameters in the VPI stream to obtain a processed image;

[0039] The receiving unit is used to receive the processed images sent from the GPU module until the processing of all images is completed;

[0040] The preset database includes a correspondence between each preset camera parameter and a preset distortion parameter corresponding to the preset camera parameter.

[0041] In a third aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the machine phase image processing method is implemented when the processor executes the program.

[0042] The present invention provides a camera image processing method, device and electronic device, which dynamically determine when the GPU executes the image processing task through multiple detection strategies, save CPU computing power, and ensure the stability of the system under various load conditions. When the CPU computing power is insufficient, by repeatedly executing the steps of processing the image on the GPU module, the advantages of the GPU in image processing are fully utilized, the image processing efficiency and speed are improved, and the image data volume is effectively reduced by extracting distortion parameters for dedistortion, cropping, scaling and other processing, combined with compression quality control, the image transmission efficiency and processing speed are improved; when the CPU computing power is sufficient, a preset image processing model is used, and the model is trained based on sample images with distortion parameter labels, which provides an accurate and efficient solution for image processing; it has important significance in the fields of autonomous driving, can effectively improve image processing performance, realize fast and stable image processing, and provide important support for the perception and decision-making capabilities of autonomous driving systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 It is a flowchart of the camera image processing method provided by the present invention;

[0045] Figure 2 is a structural schematic diagram of a camera image processing device provided by the present invention;

[0046] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] Figure 1 : is a flow chart of a camera image processing method provided by the present invention, wherein the camera image processing method comprises:

[0049] For any image among all the images, if it is determined that the current CPU module has insufficient computing power, repeat the following steps:

[0050] Step 101: Determine distortion parameters corresponding to the camera parameters from a preset database according to the camera parameters corresponding to the current image;

[0051] Step 102: Send the current image and the distortion parameters to the GPU module, instruct the GPU module to create a computer vision programming interface VPI stream according to the current image and the distortion parameters, and process the current image according to the distortion parameters in the VPI stream to obtain a processed image;

[0052] Step 103, receiving processed images sent from the GPU module until all images are processed;

[0053] The preset database includes a correspondence between each preset camera parameter and a preset distortion parameter corresponding to the preset camera parameter.

[0054] Optionally, step 101 to step 103 are loop steps, which are used to execute the above steps for each image among all the images when it is determined that the computing power of the current CPU module is insufficient. Specifically, for any image among all the images, when it is determined that the computing power of the current CPU module is insufficient, step 101 is executed. In step 101, the present invention obtains in advance the corresponding relationship between each preset camera parameter and the preset distortion parameter corresponding to the preset camera parameter, and then creates a preset database, and writes the corresponding relationship between all the preset camera parameters and the preset distortion parameter corresponding to each preset camera parameter into the preset database.

[0055] Optionally, the present invention can obtain the correspondence between the specific parameters of the camera used for the current image and the distortion parameters by querying a preset database, and then quickly determine the required distortion parameters according to the camera parameters, so as to perform subsequent image distortion processing, cropping, scaling and other operations. In other embodiments, the distortion parameters are directly given through the process of internal parameter calibration, which can be directly given by the manufacturer or calibrated by the user.

[0056] In step 102, the current image and distortion parameters are sent to the GPU module, and then the GPU module is instructed to create a computer vision programming interface (VPI) stream according to the current image and distortion parameters, and dedistortion, cropping, scaling and other operations are performed in the VPI stream according to the distortion parameters, the current image is processed, and the processed image is obtained from the GPU module. The present invention utilizes the parallel computing capability of the GPU and the mechanism of the VPI stream to achieve high efficiency and fast response of the image processing process, obtains accurate distortion parameters according to the specific parameters of the camera used by the current image, performs accurate distortion processing on the image, improves the quality and accuracy of image processing, and hands over the distortion processing task to the GPU for processing, reduces the burden on the CPU, and improves the processing performance of the overall system.

[0057] Optionally, the present invention creates a VPI stream, and the operations in the stream are all run in the GPU, and it is ensured that the operations in the stream are executed sequentially, and the operations between different streams do not affect each other by default, wherein "vpiStreamCreate" is used to create a stream, and "vpiStramDestroy" is used to destroy a stream. The operations in the stream are first defined by functions such as "vpiSubmit", and are determined by the sequence of "submit". The significance of creating a stream lies in asynchronous processing, and the program will not be blocked due to a certain step of the processing process, thereby increasing the delay. However, when synchronization is required, such as when writing from the GPU to the CPU, "vpiStreamSync" can be used to ensure that all processing programs in the stream are executed before writing.

[0058] Optionally, when performing distortion correction on the input current image, the camera internal parameters are read in through the configuration file. The parameters are generally provided by the manufacturer and generated using a pinhole model or a fisheye model. The VPI stream will generate dedistortion mapping in the X and Y directions based on the internal parameters, and load the mapping process into the stream. After execution, a dedistorted image can be obtained. The camera internal parameter matrix K is:

[0059]

[0060] Among them, f x 、f y is the focus of the camera, c x 、c y is the center point, which is usually provided by the camera manufacturer. If the camera image is scaled proportionally, then f x 、f y 、c x 、c y It also needs to be multiplied according to the scaling ratio of the width or height.

[0061] In step 103, the processed image data sent from the GPU module is received and continues to be received until all images are processed. When the GPU module completes processing of an image, the processed image data is sent to the receiving end, and the receiving end needs to continuously monitor the reception to ensure that all images can be received and processed in a timely manner.

[0062] Optionally, for any image among all the images, when it is determined that the computing power of the current CPU module is sufficient, the steps further include:

[0063] Inputting the current image into a preset image processing model to obtain a processed image output by the preset image processing model;

[0064] The preset image processing model is determined after training based on all sample images carrying distortion parameter labels and processed sample images corresponding to each sample image.

[0065] Optionally, while ensuring that the CPU computing power is sufficient, the present invention can run a preset image processing model to process images, thereby improving the speed and effect of image processing. The present invention dynamically analyzes the CPU computing power, fully utilizes the GPU for distortion processing or the CPU uses a preset image processing model, to achieve multi-faceted processing of images, and improves processing quality and efficiency. Specifically, all sample images carrying distortion parameter labels and the processed sample images corresponding to each sample image are trained to determine the preset image processing model, so that the CPU can provide more accurate and professional image processing effects when processing images, so that the processed images have higher quality, and fully utilize the CPU computing power and preset model training, which not only improves the efficiency of image processing, but also saves the use of resources.

[0066] Furthermore, the present invention can use the processed image obtained after the distortion processing by using GPU as the sample for model training, prepare a training data set with de-distortion, scaling and cropping annotations, these annotations can be about de-distortion parameters, scaling ratios, cropping positions and sizes, etc., select a suitable deep learning model architecture to implement image de-distortion, scaling and cropping processing, optionally, use Convolutional Neural Networks (CNN), use the prepared data set to train the model, during the training process, optimize the model parameters by minimizing the loss function, so that it can accurately learn the process of de-distortion, scaling and cropping images, after the training is completed, the model can also be evaluated to ensure that it performs well in tasks such as de-distortion, scaling and cropping. Optionally, use an independent test data set for evaluation, and adjust the parameters and structure of the model according to the evaluation results, and finally, deploy the trained model to the actual application of the CPU, process the current image and output the processed image.

[0067] Optionally, before determining, from a preset database according to the camera parameters corresponding to the current image, distortion parameters corresponding to the camera parameters, the method further includes:

[0068] For any processing task, if it is determined that the time for the current CPU module to execute the processing task exceeds the preset processing time, it is determined that the computing power of the current CPU module is insufficient;

[0069] Or, using a system monitoring tool to obtain the CPU module utilization, and when it is determined that the current CPU module utilization is greater than a preset utilization, determining that the current CPU module computing power is insufficient;

[0070] For any response operation, when it is determined that the response time is longer than the preset response time, it is determined that the current CPU module computing power is insufficient;

[0071] If an error message, freeze message, or crash occurs, determine that the current CPU module has insufficient computing power.

[0072] Optionally, the present invention can determine whether the current CPU module can complete the task in time by monitoring whether the processing task execution time exceeds the preset processing time. If the task execution time is too long, it may mean that the CPU computing power is insufficient; monitoring whether the CPU utilization rate is higher than the preset utilization rate can help determine whether the CPU is able to effectively process the task. High utilization rate may indicate that the CPU computing power is insufficient; if the response time exceeds the preset response time, it may indicate that the CPU processing capacity is insufficient, resulting in slow system response; when error prompts, stuck prompts or flashbacks are detected, it may be caused by insufficient CPU computing power and needs to be processed in time. The present invention can timely discover the problem of insufficient CPU computing power by real-time monitoring of the CPU execution task. According to the preset processing time, utilization rate and response time and other indicators, the system can automatically diagnose whether the CPU computing power is sufficient, so as to take corresponding measures. Optionally, in the present invention, commands such as TOP can also be used in the system to view the CPU occupancy of related processes, without waiting for obvious stuck phenomena to judge the CPU computing power. This does not affect the specific implementation of the present invention and will not be repeated here.

[0073] Optionally, the distortion parameters include radial distortion parameters and tangential distortion parameters, and processing the current image according to the distortion parameters in the VPI stream to obtain a processed image includes:

[0074] Processing the current image according to the radial distortion parameter and the tangential distortion parameter to obtain a dedistorted image;

[0075] The dedistorted image is cropped and scaled to obtain the processed image.

[0076] Optionally, the current image is processed according to the provided radial distortion parameters and tangential distortion parameters to correct the distortion in the image and make the image more realistic and accurate. After processing the radial and tangential distortions, a dedistorted image is obtained, which has corrected the distortion caused by the camera lens, making the image more consistent with the actual scene.

[0077] Furthermore, the dedistorted image is cropped and scaled, and the size and proportion of the image can be adjusted according to specific needs to meet the needs of subsequent processing or display, and finally a processed image after distortion correction, cropping and scaling is obtained. The image has been dedistorted and processed according to the set requirements, and can be used for subsequent image processing tasks or displays. The present invention can effectively improve the quality and accuracy of the image, eliminate the distortion caused by the camera lens, and crop and scale the image according to the needs, reduce the amount of transmitted data, and make the image processing efficient and stable.

[0078] Optionally, the processing the current image according to the radial distortion parameter and the tangential distortion parameter to obtain a dedistorted image includes:

[0079] x′=x(1+k1r 2 +k2r 4 +k3r 6 )+2p1xy+p2(r 2 +2x 2 ) (2)

[0080] y′=y(1+k1r 2 +k2r 4 +k3r 6 )+2p2xy+p1(r 2 +2y 2 ) (3)

[0081] Where x is the X-axis coordinate of the current image, y is the Y-axis coordinate of the current image, x′ is the X-axis coordinate of the dedistorted image, y′ is the Y-axis coordinate of the dedistorted image, k1, k2, k3 are radial distortion parameters, p1, p2 are tangential distortion parameters, r is the algebraic value,

[0082] Optionally, the distortion of the camera image is divided into radial and tangential distortions. The radial distortion is further divided into barrel and pincushion. The barrel distortion starts from the center of the image, and the radial curvature gradually increases, while the pincushion distortion gradually decreases. The mapping is:

[0083] x distorted1 =x(1+k1r 2 +k2r 4 +k3r 6 )(4)

[0084] y distorted1 =y(1+k1r 2 +k2r 4 +k3r 6 )(5)

[0085] Among them, x distorted1 is the X-axis coordinate after radial dedistortion, y distorted1 is the Y-axis coordinate after radial dedistortion. The higher-order terms of this formula will have less and less influence on the result, so some non-standard external parameter results will omit the higher-order terms, resulting in different numbers of k in the results.

[0086] In tangential distortion, this is the distortion caused by the camera lens not being parallel to the object plane, and its mapping is:

[0087] X distorted2 =x+[2p1xy+p2(r 2 +2x2 )] (6)

[0088] Y distorted2 =y+[p1(r 2 +2y 2 )+2p2xy] (7)

[0089] Among them, X distorted2 is the X-axis coordinate after tangential distortion, Y distorted2 is the Y-axis coordinate after tangential distortion.

[0090] Combining the two types of distortion to obtain their mapping relationship before and after distortion in image coordinates, the distortion parameter input provided by the camera intrinsic parameters is k1, k2, k3 and p1, p2, which need to be calculated by a special calibration program, and the driver only needs to use its results.

[0091] Optionally, the performing cropping and scaling processing on the dedistorted image to obtain the processed image includes:

[0092] Instructing the GPU module to call a preset configuration parameter NvBufferRect to crop the dedistorted image to obtain a cropped image;

[0093] The GPU module is instructed to call the NvBufsurfTransform function provided by Nvidia to scale the cropped image to obtain the processed image.

[0094] Optionally, the GPU module will call the preset configuration parameter NvBufferRect to indicate the cropping operation of the dedistorted image, and crop the dedistorted image according to the specified cropping area and parameters to obtain a cropped image. Then, the GPU module will call the NvBufsurfTransform function provided by Nvidia to scale the cropped image, and perform corresponding scaling operations on the cropped image according to the set scaling ratio or size to obtain the final processed image. Through the call of the GPU module and the above-mentioned processing steps, the cropping and scaling of the image can be realized, so as to obtain a final processed image that meets the requirements. This embodiment combines GPU hardware acceleration and the functions provided by Nvidia to efficiently process images and meet specific needs.

[0095] Optionally, after processing the current image according to the distortion parameter in the VPI stream to obtain a processed image, the method further includes:

[0096] Instruct the GPU module to call NvJpeg to compress the processed image and specify the quality score of the compressed image;

[0097] Instruct the GPU module to send the compressed image to the current CPU module.

[0098] Optionally, the present invention can also instruct the GPU module to call NvJpeg to compress the processed image. The NvJpeg library can be used to compress the image in JPEG format, and the quality score of the compressed image can be specified to balance the requirements of image quality and file size. Finally, the GPU module is instructed to send the compressed image to the current CPU module, and the processed and compressed image data is transmitted to the CPU module for subsequent processing or storage. The present invention can compress and transmit the processed image, thereby reducing the size of the image file while ensuring the image quality, and facilitating storage or transmission.

[0099] Optionally, the present invention compresses the image processed by steps 101 to 103 through NvJpeg, specifies its quality score, and directly transfers the image from the GPU space to the memory space, and finally sends it out for use by other programs. Usually, the value of the quality score is 0-100, and the specific implementation is determined by the preset JPEG compression algorithm. The higher the quality, the better the image quality, but the larger the space occupied. Conversely, the smaller the space occupied, the worse the quality. Optionally, the parameter is set to about 70-95, and the default setting is 75. Setting it too high may cause the space occupied to be too large, or even worse than using lossless compression formats such as PNG.

[0100] Optionally, before sending the current image and the distortion parameters to the GPU module, the method further includes:

[0101] For any image among all the images, convert the original color format of the image into YUV420 format;

[0102] All the images are determined in an autonomous driving scenario based on the video stream read in by the underlying driver.

[0103] Optionally, for any of all images, in the autonomous driving scenario, the original color format of the image can be converted to the YUV420 format. This color format conversion can usually effectively reduce the size of the image data and maintain the image quality, which is suitable for some video processing and transmission scenarios. Specifically, the YUV420 format can save half the space compared to the RGB format. When reading this format, the missing U and V components will be searched from adjacent rows; on the other hand, it is also the default color space for JPEG images.

[0104] Among them, all images are determined in the autonomous driving scenario based on the video stream read in by the underlying driver, that is, these image data are video stream data read from the underlying driver of the autonomous driving system, and may contain real-time driving scene information. By converting the image color format into YUV420 format and reading the image based on the video stream of the underlying driver, the image data can be made more suitable for the processing and analysis requirements in the autonomous driving scenario before being sent to the GPU module, which helps to optimize the subsequent image processing process and improve processing efficiency and quality.

[0105] The present invention provides a camera image processing method, which dynamically determines when the GPU executes the image processing task through multiple detection strategies, saves CPU computing power, and ensures the stability of the system under various load conditions. When the CPU computing power is insufficient, by repeatedly executing the steps of processing the image on the GPU module, the advantages of the GPU in image processing are fully utilized, the image processing efficiency and speed are improved, and the image data volume is effectively reduced by extracting distortion parameters for dedistortion, cropping, scaling and other processing, combined with compression quality control, the image transmission efficiency and processing speed are improved; when the CPU computing power is sufficient, a preset image processing model is used, and the model is trained based on sample images with distortion parameter labels, which provides an accurate and efficient solution for image processing; it has important significance in the fields of autonomous driving, can effectively improve image processing performance, realize fast and stable image processing, and provide important support for the perception and decision-making capabilities of autonomous driving systems.

[0106] Figure 2 is a structural schematic diagram of a camera image processing device provided by the present invention, wherein the camera image processing device comprises a repeating unit 4, wherein the repeating unit 4 comprises a determining unit 1, an indicating unit 2 and a receiving unit 3;

[0107] For any image among all the images, when it is determined that the computing power of the current CPU module is insufficient, the determination unit 1 is used to determine the distortion parameters corresponding to the camera parameters from a preset database based on the camera parameters corresponding to the current image. The working principle of the determination unit 1 can refer to the aforementioned step 101 and will not be repeated here.

[0108] The indication unit 2 is used to send the current image and the distortion parameters to the GPU module, instructing the GPU module to create a computer vision programming interface VPI stream according to the current image and the distortion parameters, and process the current image according to the distortion parameters in the VPI stream to obtain a processed image. The working principle of the indication unit 2 can refer to the aforementioned step 102 and will not be repeated here.

[0109] The receiving unit 3 is used to receive the processed images sent from the GPU module until the processing of all images is completed. The working principle of the receiving unit 3 can refer to the aforementioned step 103 and will not be repeated here.

[0110] The preset database includes a correspondence between each preset camera parameter and a preset distortion parameter corresponding to the preset camera parameter.

[0111] The present invention provides a camera image processing device, which dynamically determines when the GPU executes the image processing task through multiple detection strategies, thereby saving CPU computing power and ensuring the stability of the system under various load conditions. When the CPU computing power is insufficient, the steps of processing the image on the GPU module are repeatedly executed, so as to give full play to the advantages of the GPU in image processing and improve the image processing efficiency and speed. The distortion parameters are extracted for dedistortion, cropping, scaling and other processing, combined with compression quality control, so as to effectively reduce the image data volume and improve the image transmission efficiency and processing speed. When the CPU computing power is sufficient, a preset image processing model is used, and the model is trained based on sample images with distortion parameter labels, so as to provide an accurate and efficient solution for image processing. The device has important significance in the fields of autonomous driving, can effectively improve the image processing performance, realize fast and stable image processing, and provide important support for the perception and decision-making capabilities of the autonomous driving system.

[0112] Figure 3 Schematic diagram of the structure of the electronic device provided by the present invention. Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communication interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute the camera image processing method, which includes: for any image among all images, when it is determined that the current CPU module has insufficient computing power, repeatedly executing the following steps: according to the camera parameters corresponding to the current image, determining the distortion parameters corresponding to the camera parameters from the preset database; sending the current image and the distortion parameters to the GPU module, instructing the GPU module to create a computer vision programming interface VPI stream according to the current image and the distortion parameters, processing the current image according to the distortion parameters in the VPI stream, and obtaining a processed image; receiving the processed image sent from the GPU module until all images are processed; the preset database includes the corresponding relationship between each preset camera parameter and the preset distortion parameter corresponding to the preset camera parameter.

[0113] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0114] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute a camera image processing method provided by the above methods, the method including: for any image among all images, when it is determined that the current CPU module has insufficient computing power, repeatedly executing the following steps: determining, from a preset database, distortion parameters corresponding to the camera parameters according to the camera parameters corresponding to the current image; sending the current image and the distortion parameters to a GPU module, instructing the GPU module to create a computer vision programming interface VPI stream according to the current image and the distortion parameters, processing the current image according to the distortion parameters in the VPI stream to obtain a processed image; receiving the processed image sent by the GPU module until the processing of all images is completed; the preset database includes a correspondence between each preset camera parameter and the preset distortion parameter corresponding to the preset camera parameter.

[0115] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the camera image processing method provided by the above-mentioned methods, the method comprising: for any image among all images, when it is determined that the computing power of the current CPU module is insufficient, repeatedly executing the following steps: determining the distortion parameters corresponding to the camera parameters from a preset database according to the camera parameters corresponding to the current image; sending the current image and the distortion parameters to a GPU module, instructing the GPU module to create a computer vision programming interface VPI stream according to the current image and the distortion parameters, processing the current image according to the distortion parameters in the VPI stream to obtain a processed image; receiving the processed image sent from the GPU module until the processing of all images is completed; the preset database includes a correspondence between each preset camera parameter and the preset distortion parameter corresponding to the preset camera parameter.

[0116] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0117] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A camera image processing method, characterized in that: include: For any image among all the images, if it is determined that the current CPU module has insufficient computing power, repeat the following steps: According to the camera parameters corresponding to the current image, determining the distortion parameters corresponding to the camera parameters from a preset database; Sending the current image and the distortion parameters to the GPU module, instructing the GPU module to create a computer vision programming interface VPI stream according to the current image and the distortion parameters, and processing the current image according to the distortion parameters in the VPI stream to obtain a processed image; Receiving processed images sent from the GPU module until all images are processed; The preset database includes a correspondence between each preset camera parameter and a preset distortion parameter corresponding to the preset camera parameter.

2. The camera image processing method according to claim 1, characterized in that: For any image among all the images, when it is determined that the computing power of the current CPU module is sufficient, the steps further include: Inputting the current image into a preset image processing model to obtain a processed image output by the preset image processing model; The preset image processing model is determined after training based on all sample images carrying distortion parameter labels and processed sample images corresponding to each sample image.

3. The camera image processing method according to claim 1, characterized in that: Before determining the distortion parameters corresponding to the camera parameters from a preset database according to the camera parameters corresponding to the current image, the method further includes: For any processing task, if it is determined that the time for the current CPU module to execute the processing task exceeds the preset processing time, it is determined that the computing power of the current CPU module is insufficient; Or, using a system monitoring tool to obtain the CPU module utilization, and when it is determined that the current CPU module utilization is greater than a preset utilization, determining that the current CPU module computing power is insufficient; Or, for any response operation, when it is determined that the response time is longer than the preset response time, it is determined that the computing power of the current CPU module is insufficient; Or, if an error message, freeze message, or crash occurs, determine that the current CPU module has insufficient computing power.

4. The camera image processing method according to claim 1, characterized in that: The distortion parameters include radial distortion parameters and tangential distortion parameters, and the processing of the current image according to the distortion parameters in the VPI stream to obtain a processed image includes: Processing the current image according to the radial distortion parameter and the tangential distortion parameter to obtain a dedistorted image; The dedistorted image is cropped and scaled to obtain the processed image.

5. The camera image processing method according to claim 4, characterized in that: The step of processing the current image according to the radial distortion parameter and the tangential distortion parameter to obtain a dedistorted image includes: x′=x(1+k1r 2 +k2r 4 +k3r 6 )+2p1xy+p2(r 2 +2x 2 ) <h2 style=";text-align:left;direction:ltr">y′ = y(1+k1r<h2 style=";text-align:left;direction:ltr"> 2 <h2 style=";text-align:left;direction:ltr"> +k2r<h2 style=";text-align:left;direction:ltr"> 4 <h2 style=";text-align:left;direction:ltr"> +k3r<h2 style=";text-align:left;direction:ltr"> 6 <h2 style=";text-align:left;direction:ltr"> )+2p2xy+p1(r<h2 style=";text-align:left;direction:ltr"> 2 <h2 style=";text-align:left;direction:ltr"> +2y<h2 style=";text-align:left;direction:ltr"> 2 <h2 style=";text-align:left;direction:ltr"> ) Where x is the X-axis coordinate of the current image, y is the Y-axis coordinate of the current image, x′ is the X-axis coordinate of the dedistorted image, y′ is the Y-axis coordinate of the dedistorted image, k1, k2, k3 are radial distortion parameters, p1, p2 are tangential distortion parameters, r is the algebraic value, 6. The camera image processing method according to claim 4, characterized in that: The step of performing cropping and scaling processing on the dedistorted image to obtain the processed image includes: Instructing the GPU module to call a preset configuration parameter NvBufferRect to crop the dedistorted image to obtain a cropped image; The GPU module is instructed to call the NvBufsurfTransform function provided by Nvidia to scale the cropped image to obtain the processed image.

7. The camera image processing method according to claim 1, characterized in that: After processing the current image according to the distortion parameter in the VPI stream to obtain a processed image, the method further includes: Instruct the GPU module to call NvJpeg to compress the processed image and specify the quality score of the compressed image; Instruct the GPU module to send the compressed image to the current CPU module.

8. The camera image processing method according to claim 1, characterized in that: Before sending the current image and the distortion parameters to the GPU module, the method further includes: For any image among all the images, convert the original color format of the image into YUV420 format; All the images are determined in an autonomous driving scenario based on the video stream read in by the underlying driver.

9. A camera image processing device, characterized in that: comprising a repeating unit, wherein the repeating unit comprises a determining unit, an indicating unit and a receiving unit; For any image among all the images, when it is determined that the computing power of the current CPU module is insufficient, the determining unit is used to determine the distortion parameters corresponding to the camera parameters from a preset database according to the camera parameters corresponding to the current image; The instructing unit is used to send the current image and the distortion parameters to the GPU module, instruct the GPU module to create a computer vision programming interface VPI stream according to the current image and the distortion parameters, and process the current image according to the distortion parameters in the VPI stream to obtain a processed image; The receiving unit is used to receive the processed images sent from the GPU module until the processing of all images is completed; The preset database includes a correspondence between each preset camera parameter and a preset distortion parameter corresponding to the preset camera parameter.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the camera image processing method according to any one of claims 1 to 8 is implemented.

Images