Projection effect evaluation methods, equipment, storage media and software products

Abstract

This application provides a method, device, storage medium, and program product for judging projection effects. The method acquires projection data and image data of a projection area. The projection data is obtained by projecting point cloud data onto the projection area, and the image data is image data captured in the projection area. The projection data and image data are processed in grayscale to obtain an image grayscale image and a projection grayscale image. The image grayscale image and the projection grayscale image are input into a preset neural network to obtain projection labels corresponding to the projection data. The preset neural network is trained using multiple image grayscale image samples, multiple projection grayscale image samples, and multiple preset projection labels. The projection effect is judged based on the projection labels, eliminating the need for manual identification of the difference between the projection data and the image data. Through the preset neural network, the accuracy of the projection can be judged efficiently and quickly, improving the efficiency of projection effect judgment.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a method, device, storage medium, and program product for judging projection effects. Background Technology

[0002] In recent years, automobile companies and the Internet industry have launched research and development of intelligent connected vehicles. The most core technology is autonomous driving technology. Autonomous driving technology is a key technology that the automotive and transportation industries need to break through in order to compete in the future market. The implementation of autonomous driving technology faces many challenges. To ensure the safety of autonomous driving, multi-sensor fusion is required, which combines point cloud data and image data to obtain richer information by leveraging the advantages of each data and providing reasonable driving plans for autonomous driving. Among them, the most common method for combining point cloud data and image data is to project point cloud data onto an image surface.

[0003] However, due to sensor jitter and inaccurate calibration, point clouds may be offset when projected onto the image plane. This projection offset can significantly interfere with subsequent processing, leading to inaccurate projection. Therefore, it is necessary to assess the accuracy of the projection. Related technologies typically employ manual identification methods, overlaying the point cloud projection rendering with the corresponding camera-captured images. The overlaid images are then manually filtered to determine the accuracy of the projection result.

[0004] However, in existing technologies, manual identification and screening methods are inefficient and make it difficult to efficiently determine whether the point cloud projection results are accurate. Summary of the Invention

[0005] This application provides a method, device, storage medium, and program product for judging projection effects, thereby solving the technical problem that in the prior art, the manual identification and screening method is inefficient and it is difficult to efficiently judge whether the point cloud projection results are accurate.

[0006] Firstly, this application provides a method for judging projection effects, including:

[0007] The projection data and image data of the projection area are obtained, wherein the projection data is obtained by projecting point cloud data onto the projection area, and the image data is image data captured in the projection area;

[0008] The projection data and the image data are processed to obtain an image grayscale image and a projection grayscale image;

[0009] The image grayscale image and the projected grayscale image are input into a preset neural network to obtain the projection label corresponding to the projection data. The preset neural network is trained by multiple image grayscale image samples, multiple projected grayscale image samples and multiple preset projection labels.

[0010] The effect of the projection is determined based on the projection label.

[0011] In this embodiment, to determine the accuracy of the projection effect, projection data and image data of the projection area are first acquired. The projection data is the projection of point cloud data onto the projection area, and the image data is the captured image of the projection area. Grayscale processing is performed on the projection data and image data to obtain image grayscale and projection grayscale, which facilitates processing of different data. The image grayscale and projection grayscale are then input into a preset neural network. Through this preset neural network, projection labels corresponding to the projection data can be output. The projection effect can be judged based on the projection labels, and the accuracy of the projection can be determined. In this embodiment, there is no need for manual identification of the difference between projection data and image data. Through the preset neural network, the accuracy of the projection can be judged efficiently and quickly, improving the efficiency of projection effect judgment.

[0012] Optionally, before inputting the image grayscale image and the projected grayscale image into a preset neural network, the method further includes:

[0013] Acquire multiple image grayscale samples, multiple projected grayscale samples, and multiple preset projection labels;

[0014] The multiple image grayscale samples, multiple projected grayscale samples, and multiple preset projection labels are input into the neural network for training to obtain the preset neural network.

[0015] Here, in this embodiment of the application, a neural network can be trained using multiple image grayscale samples, multiple projected grayscale samples, and multiple preset projection labels to obtain a preset neural network whose input is an image grayscale image and a projected grayscale image, and whose output is the projection label corresponding to the projection data. By training with multiple image grayscale samples, multiple projected grayscale samples, and multiple preset projection labels, the accuracy of the output result of the preset neural network is improved, and the accuracy of the projection effect judgment is further improved.

[0016] Optionally, after performing grayscale processing on the projection data and the image data to obtain the image grayscale image and the projection grayscale image, the method further includes:

[0017] The projected grayscale image is subjected to pixel filling processing to obtain a dense projected grayscale image;

[0018] Accordingly, the image grayscale image and the projected grayscale image are input into a preset neural network to obtain the projection label corresponding to the projection data. The preset neural network is trained using multiple image grayscale image samples, multiple projected grayscale image samples, and multiple preset projection labels, including:

[0019] The image grayscale image, the projected grayscale image, and the dense projected grayscale image are input into a preset neural network to obtain the projection label corresponding to the projection data. The preset neural network is trained using multiple image grayscale image samples, multiple projected grayscale image samples, multiple dense projected grayscale image samples, and multiple preset projection labels.

[0020] Here, in this embodiment of the application, after performing grayscale processing on the projection data and image data to obtain the image grayscale image and the projection grayscale image, pixel filling processing is performed on the projection grayscale image. Since the original projection data and projection grayscale image are sparse and the image has many holes, while the features of the real image are dense, in order to ensure the accuracy of data processing, pixel filling is used to process the projection data to obtain a dense projection grayscale image. Inputting the above-mentioned dense projection grayscale image into a preset neural network can obtain a more accurate projection label, further improving the accuracy of projection effect judgment.

[0021] Optionally, after performing pixel filling processing on the projected grayscale image to obtain a dense projected grayscale image, the method further includes:

[0022] The difference between the dense projection grayscale image and the image grayscale image is obtained to obtain the difference grayscale image.

[0023] Accordingly, the image grayscale image and the projected grayscale image are input into a preset neural network to obtain the projection label corresponding to the projection data. The preset neural network is trained using multiple image grayscale image samples, multiple projected grayscale image samples, and multiple preset projection labels, including:

[0024] The image grayscale image, the projected grayscale image, the dense projected grayscale image, and the difference grayscale image are input into a preset neural network to obtain the projection label corresponding to the projection data. The preset neural network is trained using multiple image grayscale image samples, multiple projected grayscale image samples, multiple dense projected grayscale image samples, multiple difference grayscale image samples, and multiple preset projection labels.

[0025] Here, in this embodiment of the application, the difference between the image grayscale image and the dense projection grayscale image can be obtained to obtain a difference grayscale image. The difference grayscale image is also input into a preset neural network. During training, the preset neural network combines the difference between the image grayscale image sample and the dense projection grayscale image sample. Through this difference, the offset between the projected image and the original image can be accurately reflected. Using this difference as training data for the preset neural network can improve the accuracy of the preset neural network and further improve the accuracy of the projection effect judgment.

[0026] Optionally, the preset projection labels include good labels and poor labels;

[0027] Accordingly, judging the projection effect based on the projection label includes:

[0028] If the projection label corresponding to the projection data is a good label, then the projection effect is judged to be accurate;

[0029] If the projection label corresponding to the projection data is a substandard label, then the projection effect is judged to be inaccurate.

[0030] Here, in this embodiment of the application, the projection labels are divided into good labels and poor labels. By using a binary classification method, the projection effect can be effectively distinguished. The process is simple, the result is clear, and the projection effect can be judged accurately and efficiently.

[0031] Optionally, the preset projection labels include good labels, top-biased labels, and bottom-biased labels;

[0032] Accordingly, judging the projection effect based on the projection label includes:

[0033] If the projection label corresponding to the projection data is a good label, then the projection effect is judged to be accurate;

[0034] If the projection label corresponding to the projection data is a label that is too high, then the projection effect is judged to be inaccurate and the projection is too high.

[0035] If the projection label corresponding to the projection data is a lower label, then the projection effect is judged to be inaccurate and the projection is lower.

[0036] Here, in this embodiment of the application, the projection labels are divided into good labels, top labels, and bottom labels. This not only accurately judges the quality of the projection effect, but also determines whether the projection is offset up or down relative to the original image based on the top and bottom labels. The output prediction results are richer, further improving the accuracy of the projection effect judgment and the practicality of the projection prediction, making it easier for users to better understand the projection situation.

[0037] Optionally, after determining the projection effect based on the projection label, the method further includes:

[0038] Adjust the projection based on the projection effect.

[0039] Here, in this embodiment of the application, the projection can be adjusted according to the projection effect. If it is determined that the projection is inaccurate, the projection can be adjusted. Furthermore, if it is determined that the projection is too high or too low, targeted adjustments can be made according to the judgment result to improve the accuracy and quality of the projection, thereby further ensuring the safety of autonomous vehicles.

[0040] Secondly, embodiments of this application provide a projection effect judging device, comprising:

[0041] The first acquisition module is used to acquire projection data and image data of the projection area, wherein the projection data is obtained by projecting point cloud data onto the projection area, and the image data is image data captured in the projection area;

[0042] The first processing module is used to perform grayscale processing on the projection data and the image data to obtain an image grayscale image and a projection grayscale image;

[0043] An input module is used to input the image grayscale image and the projected grayscale image into a preset neural network to obtain the projection label corresponding to the projection data. The preset neural network is trained by multiple image grayscale image samples, multiple projected grayscale image samples and multiple preset projection labels.

[0044] The judgment module is used to judge the effect of the projection based on the projection label.

[0045] Optionally, before the input module inputs the image grayscale image and the projected grayscale image into the preset neural network, the above-mentioned device further includes:

[0046] The second acquisition module is used to acquire multiple image grayscale samples, multiple projected grayscale samples, and multiple preset projection labels;

[0047] The training module is used to input the multiple image grayscale samples, multiple projected grayscale samples and multiple preset projection labels into the neural network for training, so as to obtain the preset neural network.

[0048] Optionally, after the first processing module performs grayscale processing on the projection data and the image data to obtain an image grayscale image and a projection grayscale image, the above apparatus further includes:

[0049] The second processing module is used to perform pixel filling processing on the projected grayscale image to obtain a dense projected grayscale image.

[0050] Accordingly, the input module is specifically used to input the image grayscale image, the projected grayscale image, and the dense projected grayscale image into a preset neural network to obtain the projection label corresponding to the projection data. The preset neural network is trained by multiple image grayscale image samples, multiple projected grayscale image samples, multiple dense projected grayscale image samples, and multiple preset projection labels.

[0051] Optionally, after the second processing module performs pixel filling processing on the projected grayscale image to obtain a dense projected grayscale image, the above apparatus further includes:

[0052] The third processing module is used to subtract the dense projection grayscale image from the image grayscale image to obtain a difference grayscale image.

[0053] Accordingly, the input module is specifically used to input the image grayscale image, the projected grayscale image, the dense projected grayscale image, and the difference grayscale image into a preset neural network to obtain the projection label corresponding to the projection data. The preset neural network is trained using multiple image grayscale image samples, multiple projected grayscale image samples, multiple dense projected grayscale image samples, multiple difference grayscale image samples, and multiple preset projection labels.

[0054] Optionally, the preset projection labels include good labels and poor labels;

[0055] Accordingly, the judgment module is specifically used to determine that the projection effect is accurate if the projection label corresponding to the projection data is a good label;

[0056] If the projection label corresponding to the projection data is a substandard label, then the projection effect is judged to be inaccurate.

[0057] Optionally, the preset projection labels include good labels, top-biased labels, and bottom-biased labels;

[0058] Accordingly, the judgment module is specifically used to determine that the projection effect is accurate if the projection label corresponding to the projection data is a good label;

[0059] If the projection label corresponding to the projection data is a label that is too high, then the projection effect is judged to be inaccurate and the projection is too high.

[0060] If the projection label corresponding to the projection data is a lower label, then the projection effect is judged to be inaccurate and the projection is lower.

[0061] Optionally, after the judgment module judges the projection effect based on the projection label, the above device further includes:

[0062] The adjustment module is used to adjust the projection according to the projection effect.

[0063] Thirdly, embodiments of this application provide a projection effect judging device, including: at least one processor and a memory;

[0064] The memory stores computer-executed instructions;

[0065] The at least one processor executes the computer execution instructions stored in the memory, causing the at least one processor to perform the projection effect judgment method as described in the first aspect and various possible designs of the first aspect.

[0066] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing computer-executable instructions. When a processor executes the computer-executable instructions, it implements the projection effect judgment method described in the first aspect and various possible designs of the first aspect.

[0067] Fifthly, embodiments of the present invention provide a computer program product, including a computer program, which, when executed by a processor, implements the projection effect judgment method as described in the first aspect and various possible designs of the first aspect.

[0068] The projection effect judgment method, device, storage medium, and program product provided in this application embodiment, wherein the method, in order to determine the accuracy of the projection effect, first acquires projection data and image data of the projection area. Here, the projection data is the projection of point cloud data onto the projection area, and the image data is the image of the projection area captured by photography. Grayscale processing is performed on the projection data and image data to obtain image grayscale image and projection grayscale image, thereby facilitating processing for different data. Then, the image grayscale image and projection grayscale image are input into a preset neural network. Through the preset neural network, the projection label corresponding to the projection data can be output. The projection effect can be judged based on the projection label, and the accuracy of the projection can be determined. This application embodiment does not require manual identification of the difference between projection data and image data. Through the preset neural network, the accuracy of the projection can be judged efficiently and quickly, improving the efficiency of projection effect judgment. Attached Figure Description

[0069] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0070] Figure 1 A schematic diagram illustrating the superposition of a projected image and an original image, provided as an embodiment of this application;

[0071] Figure 2 This is a schematic diagram of a projection effect judgment system architecture provided in an embodiment of this application;

[0072] Figure 3 A flowchart illustrating a projection effect judgment method provided in an embodiment of this application;

[0073] Figure 4 A flowchart illustrating another projection effect judgment method provided in an embodiment of this application;

[0074] Figure 5 A flowchart illustrating another projection effect judgment method provided in an embodiment of this application;

[0075] Figure 6 This is a schematic diagram of a projection effect judging device provided in an embodiment of this application;

[0076] Figure 7 This is a schematic diagram of a projection effect judging device provided in an embodiment of this application. Detailed Implementation

[0077] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0078] The terms “first,” “second,” “third,” and “fourth,” etc. (if present), in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0079] In recent years, automotive companies and internet companies have launched research and development of intelligent connected vehicles, with autonomous driving being the core technology. Autonomous driving is a key technology that the automotive and transportation industries need to overcome to compete in the future market. The implementation of autonomous driving technology faces numerous challenges. To ensure the safety of autonomous driving, multi-sensor fusion is required, combining point cloud data and image data. Leveraging the advantages of each data source yields richer information, providing rational driving plans for autonomous driving. Point cloud data refers to data acquired by sensors such as LiDAR, recorded in the form of three-dimensional points. Each point contains three-dimensional coordinates and precise positional information. Image data refers to images captured by sensors such as cameras, generated as bitmaps composed of pixel arrays, containing rich information such as color and texture. The most common method for combining point cloud data and image data is to project the point cloud data onto an image surface.

[0080] However, due to sensor jitter and inaccurate calibration, point clouds may be offset when projected onto the image plane. This projection offset can significantly interfere with subsequent processing, causing inaccurate projection. For example, in automated map creation processes, point clouds are projected onto the image plane, and then the point clouds are assigned the color information of the corresponding image pixels. If the projection is inaccurate, the generated results will be problematic. Figure 1 As shown, Figure 1 This is a schematic diagram illustrating the superposition of a projected image and an original image, as provided in an embodiment of this application. Figure 1 As shown, the projected image is composed of discontinuous points and is offset from the original image. Therefore, we need to identify and discard these inaccurate and offset data. The problem we need to solve is how to efficiently evaluate the accuracy of the projection. Therefore, we need to judge the accuracy of the projection. In related technologies, a manual identification method is usually used, which involves overlaying the point cloud projection rendering image with the corresponding image captured by the camera, and then manually filtering the overlaid image to determine the accuracy of the projection result.

[0081] However, existing technologies suffer from the problem that manual identification and screening methods are inefficient and make it difficult to efficiently determine whether the point cloud projection results are accurate.

[0082] To address the aforementioned technical problems, this application provides a method, device, storage medium, and program product for judging projection effects. First, projection data and image data of the projection area are acquired. Grayscale processing is performed on the projection data and image data to obtain an image grayscale image and a projection grayscale image, facilitating processing of different data. Then, the image grayscale image and projection grayscale image are input into a preset neural network. This preset neural network outputs projection labels corresponding to the projection data, allowing for the judgment of the projection effect and determining whether the projection is accurate. This application eliminates the need for manual identification of the difference between projection data and image data. The preset neural network efficiently and quickly determines whether the projection is accurate, improving the efficiency of projection effect judgment.

[0083] Optional, Figure 2 This is a schematic diagram of a projection effect judgment system architecture provided in an embodiment of this application. Figure 2 In the above architecture, at least one of receiving device 201, processor 202 and display device 203 is included.

[0084] It is understood that the structure illustrated in the embodiments of this application does not constitute a specific limitation on the architecture of the projection effect judgment system. In other feasible embodiments of this application, the above architecture may include more or fewer components than illustrated, or combine some components, or split some components, or arrange different components, which can be determined according to the actual application scenario and is not limited here. Figure 2 The components shown can be implemented in hardware, software, or a combination of both.

[0085] In the specific implementation process, the receiving device 201 can be an input / output interface or a communication interface.

[0086] The processor 202 can acquire projection data and image data of the projection area, perform grayscale processing on the projection data and image data to obtain image grayscale images and projection grayscale images, thus facilitating processing of different data. The image grayscale images and projection grayscale images are then input into a preset neural network. Through this preset neural network, projection labels corresponding to the projection data can be output. The projection effect can be judged based on the projection labels, and the accuracy of the projection can be determined. In this embodiment, there is no need for manual identification of the difference between projection data and image data. Through the preset neural network, the accuracy of the projection can be judged efficiently and quickly, improving the efficiency of judging the projection effect.

[0087] The display device 203 can be used to display the above results, etc.

[0088] The display device can also be a touch screen, used to receive user commands while displaying the above content, so as to achieve interaction with the user.

[0089] It should be understood that the aforementioned processor can be implemented by reading instructions from memory and executing those instructions, or it can be implemented through chip circuitry.

[0090] Furthermore, the network architecture and business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0091] The technical solution of this application will be described in detail below with reference to specific embodiments:

[0092] Figure 3 This is a flowchart illustrating a projection effect judgment method provided in an embodiment of this application. The execution entity of this embodiment can be... Figure 2 The processor 202 in the code can be used to execute tasks based on the specific application scenario. For example... Figure 3 As shown, the method includes the following steps:

[0093] S301: Obtain projection data and image data of the projection area.

[0094] The projection data is obtained by projecting point cloud data onto the projection area, and the image data is the image data captured in the projection area.

[0095] Optionally, point cloud data is acquired by sensors such as LiDAR, and the projection data of the projection area is the projection of point cloud data onto the projection area. The projection data is the projection image. In order to determine the effect of the projection image, the image data of the projection area of ​​the projection image is acquired.

[0096] Optionally, the image data can be acquired by sensors such as cameras, or by acquiring map image data of a specified area.

[0097] Optionally, after acquiring the projection data and image data, the projection data and image data can be preprocessed so that the data can be input into a preset neural network for processing.

[0098] S302: Perform grayscale processing on the projection data and image data to obtain the image grayscale image and the projection grayscale image.

[0099] Optionally, grayscale processing of an image can be performed by directly calling a function, or by using the maximum value method, component method, average value method, and weighted average method.

[0100] By processing the grayscale of an image, it is easier to perform subsequent processing on the image data, which helps to improve the accuracy of the data.

[0101] S303: Input the image grayscale and the projected grayscale into a preset neural network to obtain the projection label corresponding to the projection data.

[0102] The preset neural network is trained using multiple image grayscale samples, multiple projected grayscale samples, and multiple preset projection labels.

[0103] Optionally, before inputting the image grayscale and projected grayscale into the preset neural network, the neural network can be pre-trained to obtain the preset neural network. Specifically:

[0104] Acquire multiple image grayscale samples, multiple projected grayscale samples, and multiple preset projection labels;

[0105] Multiple image grayscale samples, multiple projected grayscale samples, and multiple preset projection labels are input into a neural network for training to obtain a preset neural network.

[0106] Optionally, the default neural network is the ResNet18 network structure.

[0107] As an example, a neural network training method is as follows: if the projected labels are poor labels and good labels, the training process is the same as the classic deep learning model training process. First, the neural network is initialized, and then the network reads in the data to perform forward inference to obtain the predicted good labels or poor labels.

[0108] Optionally, after obtaining the predicted value, the predicted value can be compared with the label to calculate the error. If the error is higher than a threshold, the network parameters are updated using the Error Back Propagation (BP) algorithm, and the training is iterated for the next iteration. If the error is lower than the training error threshold, the training is complete, and the trained model, which is the preset neural network, is output. Here, the label is the label corresponding to the manually identified training data. It can be understood that the training error threshold here can be determined according to the actual situation, and this application embodiment does not impose specific limitations on it.

[0109] Optionally, after obtaining the predicted value, the network parameters can be updated, and the next training iteration can be performed. When the number of training iterations reaches a preset training iteration threshold, training is stopped, and the trained model, which is a preset neural network, is output. It is understood that the preset training iteration threshold can be determined according to the actual situation, and this application embodiment does not impose specific restrictions on it.

[0110] Here, in this embodiment of the application, a neural network can be trained using multiple image grayscale samples, multiple projected grayscale samples, and multiple preset projection labels to obtain a preset neural network whose input is an image grayscale image and a projected grayscale image, and whose output is the projection label corresponding to the projection data. By training with multiple image grayscale samples, multiple projected grayscale samples, and multiple preset projection labels, the accuracy of the output result of the preset neural network is improved, and the accuracy of the projection effect judgment is further improved.

[0111] S304: Determine the projection effect based on the projection label.

[0112] Optionally, the projection result can be determined based on different projection labels.

[0113] One feasible implementation is to pre-define projection labels including good labels and bad labels;

[0114] Accordingly, judging the projection effect based on the projection label includes:

[0115] If the projection label corresponding to the projection data is a good label, then the projection effect is judged to be accurate;

[0116] If the projection label corresponding to the projection data is a low-quality label, then the projection effect is inaccurate.

[0117] Here, in this embodiment of the application, the projection labels are divided into good labels and poor labels. By using a binary classification method, the projection effect can be effectively distinguished. The process is simple, the result is clear, and the projection effect can be judged accurately and efficiently.

[0118] Another feasible implementation is to pre-define projection labels, including good labels, top-biased labels, and bottom-biased labels;

[0119] Accordingly, judging the projection effect based on the projection label includes:

[0120] If the projection label corresponding to the projection data is a good label, then the projection effect is judged to be accurate;

[0121] If the projection label corresponding to the projection data is an up label, then the projection effect is inaccurate and the projection is too high.

[0122] If the projection label corresponding to the projection data is a down label, then the projection effect is inaccurate and the projection is too low.

[0123] Here, in this embodiment of the application, the projection labels are divided into good labels, top labels, and bottom labels. This not only accurately judges the quality of the projection effect, but also determines whether the projection is offset up or down relative to the original image based on the top and bottom labels. The output prediction results are richer, further improving the accuracy of the projection effect judgment and the practicality of the projection prediction, making it easier for users to better understand the projection situation.

[0124] Another feasible implementation is that the projection label includes multiple scores, for example, 60, 70, 80, 90, 100, etc. Scores can be assigned based on the projection label. Correspondingly, if the score is greater than a preset score threshold, the projection effect is considered accurate; if it is less than the preset score threshold, the projection effect is considered inaccurate. It is understood that the setting of the scores in the projection label and the preset score threshold can be determined according to the actual situation, and this application does not impose specific restrictions on them.

[0125] In one possible implementation, the projection can also be adjusted based on the projection effect. After judging the projection effect based on the projection label, the method further includes: adjusting the projection based on the projection effect.

[0126] Here, in this embodiment of the application, the projection can be adjusted according to the projection effect. If it is determined that the projection is inaccurate, the projection can be adjusted. Furthermore, if it is determined that the projection is too high or too low, targeted adjustments can be made according to the judgment result to improve the accuracy and quality of the projection, thereby further ensuring the safety of autonomous vehicles.

[0127] In order to determine the accuracy of the projection effect, this embodiment first acquires projection data and image data of the projection area. Here, the projection data is the projection of point cloud data onto the projection area, and the image data is the captured image of the projection area. Grayscale processing is performed on the projection data and image data to obtain image grayscale and projection grayscale, which facilitates processing for different data. Then, the image grayscale and projection grayscale are input into a preset neural network. Through the preset neural network, the projection label corresponding to the projection data can be output. The projection effect can be judged based on the projection label, and the accuracy of the projection can be determined. This embodiment does not require manual identification of the difference between projection data and image data. Through the preset neural network, the accuracy of the projection can be determined efficiently and quickly, improving the efficiency of projection effect judgment.

[0128] Optionally, embodiments of this application may further process the projection data to fill pixels, thereby improving the accuracy of projection effect judgment. Figure 4 A flowchart illustrating another projection effect judgment method provided in this application embodiment is shown below. Figure 4 As shown, the method includes:

[0129] S401: Obtain projection data and image data of the projection area.

[0130] S402: Perform grayscale processing on the projection data and image data to obtain the image grayscale image and the projection grayscale image.

[0131] The implementation methods of steps S401-S402 are the same as those of steps S301-S302, and will not be described in detail here.

[0132] S403: Perform pixel filling processing on the projected grayscale image to obtain a dense projected grayscale image.

[0133] Since a projected grayscale image is the result of projecting a point cloud onto an image, this projected image is sparse and has many holes, while the basic features of a real image are dense. Therefore, such an image cannot be directly processed using common image processing algorithms, so pixel filling processing can be performed in advance.

[0134] Optionally, the specific method for pixel filling processing is as follows:

[0135] Define a kernel, denoted by size n. Based on the characteristics of point cloud projection maps, the point cloud is sparser towards the bottom and denser towards the top. Therefore, the formula for n is as follows:

[0136]

[0137] Where N max N represents the maximum kernel size. min Let N represent the minimum kernel size, H represent the maximum number of rows in the projection graph, and h represent the current row number. This ensures that the kernel size is smaller as the row number increases, and larger as the row number increases. For example, if the projection graph has a row height of 480, and N is set to N, then N = 0. max For 10, N min If the value is 2, then in the first row where h = 0, n = 2, and in the last row where h = H = 480, n = 10.

[0138] With a kernel, each pixel undergoes an expansion operation using its corresponding kernel. For example, if the kernel size of the first row is 2, then for each pixel in the first row, if there are non-zero values ​​in the positions of the two surrounding pixels (i.e., white), then the value of that pixel is set to the maximum value of those surrounding pixels.

[0139] By performing this operation, the holes in the projection map can be filled in as much as possible. Moreover, by using this strategy, a small kernel size can be used for distant point clouds and a large kernel size can be used for sparse point clouds nearby, thus obtaining a better projection map without holes.

[0140] S404: Input the image grayscale image, the projected grayscale image, and the dense projected grayscale image into a preset neural network to obtain the projection label corresponding to the projection data.

[0141] The preset neural network is trained using multiple image grayscale samples, multiple projected grayscale samples, multiple dense projected grayscale samples, and multiple preset projection labels.

[0142] Optionally, it can also be combined with image recognition capabilities to perform semantic segmentation and recognition on the image and the densed point cloud projection map, such as recognizing lane lines and other elements. The results of both recognitions can then be fed into the network to learn features better and improve the accuracy of projection effect judgment.

[0143] S405: Determine the projection effect based on the projection label.

[0144] In this embodiment, after performing grayscale processing on the projection data and image data to obtain the image grayscale image and the projection grayscale image, pixel filling processing is performed on the projection grayscale image. Since the original projection data and projection grayscale image are sparse and the image has many holes, while the features of the real image are dense, in order to ensure the accuracy of data processing, pixel filling is used to process the projection data into a dense projection grayscale image. Inputting the above dense projection grayscale image into a preset neural network can obtain more accurate projection labels, further improving the accuracy of projection effect judgment.

[0145] Optionally, embodiments of this application may also calculate the difference between the image data and the processed projection data, thereby improving the accuracy of the projection effect judgment. Correspondingly, Figure 5 A flowchart illustrating another projection effect judgment method provided in this application embodiment is shown below. Figure 5 As shown, the method includes:

[0146] S501: Obtain projection data and image data of the projection area.

[0147] S502: Perform grayscale processing on the projection data and image data to obtain the image grayscale image and the projection grayscale image.

[0148] S503: Perform pixel filling processing on the projected grayscale image to obtain a dense projected grayscale image.

[0149] The implementation methods of steps S501-S503 are the same as those of steps S401-S403, and will not be described in detail here.

[0150] S504: Subtract the dense projection grayscale image from the image grayscale image to obtain the difference grayscale image.

[0151] S505: Input the image grayscale image, the projected grayscale image, the dense projected grayscale image, and the difference grayscale image into the preset neural network to obtain the projection label corresponding to the projection data.

[0152] The preset neural network is trained using multiple image grayscale samples, multiple projected grayscale samples, multiple dense projected grayscale samples, multiple difference grayscale samples, and multiple preset projection labels.

[0153] S506: Determine the projection effect based on the projection label.

[0154] In this embodiment, the difference between the image grayscale image and the dense projection grayscale image can be obtained to obtain a difference grayscale image. The difference grayscale image is also input into a preset neural network. During training, the preset neural network combines the difference between the image grayscale image sample and the dense projection grayscale image sample, which can improve the accuracy of the preset neural network and further improve the accuracy of the projection effect judgment.

[0155] Figure 6 This is a schematic diagram of a projection effect judging device provided in an embodiment of this application, as shown below. Figure 6 As shown, the apparatus in this embodiment includes: a first acquisition module 601, a first processing module 602, an input module 603, and a judgment module 604. The projection effect judgment device here can be the processor 202 itself, or a chip or integrated circuit that implements the functions of the processor 202. It should be noted that the division of the first acquisition module 601, the first processing module 602, the input module 603, and the judgment module 604 is only a logical functional division; physically, they can be integrated or independent.

[0156] The first acquisition module 601 is used to acquire projection data and image data of the projection area. The projection data is obtained by projecting point cloud data onto the projection area, and the image data is image data captured in the projection area.

[0157] The first processing module 602 is used to perform grayscale processing on the projection data and image data to obtain the image grayscale image and the projection grayscale image;

[0158] The input module 603 is used to input the image grayscale image and the projected grayscale image into a preset neural network to obtain the projection label corresponding to the projection data. The preset neural network is trained by multiple image grayscale image samples, multiple projected grayscale image samples and multiple preset projection labels.

[0159] The judgment module 604 is used to judge the effect of the projection based on the projection label.

[0160] Optionally, before the input module 603 inputs the image grayscale image and the projected grayscale image into the preset neural network, the above-mentioned device further includes:

[0161] The second acquisition module is used to acquire multiple image grayscale samples, multiple projected grayscale samples, and multiple preset projection labels;

[0162] The training module is used to input multiple image grayscale samples, multiple projected grayscale samples, and multiple preset projection labels into the neural network for training, thereby obtaining the preset neural network.

[0163] Optionally, after the first processing module 602 performs grayscale processing on the projection data and image data to obtain the image grayscale image and the projection grayscale image, the above-mentioned device further includes:

[0164] The second processing module is used to perform pixel filling processing on the projected grayscale image to obtain a dense projected grayscale image.

[0165] Correspondingly, the input module 603 is specifically used to input the image grayscale image, the projected grayscale image, and the dense projected grayscale image into a preset neural network to obtain the projection label corresponding to the projection data. The preset neural network is trained by multiple image grayscale image samples, multiple projected grayscale image samples, multiple dense projected grayscale image samples, and multiple preset projection labels.

[0166] Optionally, after the second processing module performs pixel filling processing on the projected grayscale image to obtain a dense projected grayscale image, the above apparatus further includes:

[0167] The third processing module is used to subtract the dense projection grayscale image from the image grayscale image to obtain the difference grayscale image.

[0168] Correspondingly, the input module 603 is specifically used to input the image grayscale image, the projected grayscale image, the dense projected grayscale image, and the difference grayscale image into a preset neural network to obtain the projection label corresponding to the projection data. The preset neural network is trained by multiple image grayscale image samples, multiple projected grayscale image samples, multiple dense projected grayscale image samples, multiple difference grayscale image samples, and multiple preset projection labels.

[0169] Optionally, the preset projection labels include good labels and poor labels;

[0170] Accordingly, the judgment module 604 is specifically used to judge the projection effect as accurate if the projection label corresponding to the projection data is a good label;

[0171] If the projection label corresponding to the projection data is a low-quality label, then the projection effect is inaccurate.

[0172] Optionally, the preset projection labels include good labels, top labels, and bottom labels;

[0173] Accordingly, the judgment module 604 is specifically used to judge the projection effect as accurate if the projection label corresponding to the projection data is a good label;

[0174] If the projection label corresponding to the projection data is a label that is too high, then the projection effect is inaccurate and the projection is too high.

[0175] If the projection label corresponding to the projection data is a lower label, then the projection effect is inaccurate and the projection is lower.

[0176] Optionally, after the judgment module 604 judges the projection effect based on the projection label, the above device further includes:

[0177] The adjustment module is used to adjust the projection based on the projection effect.

[0178] Figure 7 This is a schematic diagram of a projection effect judging device provided in an embodiment of this application. The components shown herein, their connections and relationships, and their functions are merely examples and do not limit the implementation of this application described and / or claimed herein.

[0179] like Figure 7 As shown, the projection effect judging device includes a processor 701 and a memory 702. The various components are interconnected via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processor 701 can process instructions executed within the projection effect judging device, including instructions stored in or on the memory for displaying graphic information on an external input / output device (such as a display device coupled to an interface). In other embodiments, multiple processors and / or multiple buses can be used with multiple memories or multiple memory sets, if desired. Figure 7 Take the 701 processor as an example.

[0180] Memory 702, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the method of the projection effect judgment device in the embodiments of this application (e.g., attached). Figure 6The first acquisition module 601, the first processing module 602, the input module 603, and the judgment module 604 are shown. The processor 701 executes various functional applications and data processing of the server by running non-transient software programs, instructions, and modules stored in the memory 702, thereby realizing the method of the projection effect judgment device in the above method embodiment.

[0181] The projection effect evaluation device may further include an input device 703 and an output device 704. The processor 701, memory 702, input device 703, and output device 704 can be connected via a bus or other means. Figure 7 Taking the example of a connection between China and Israel via a bus.

[0182] Input device 703 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the projection effect judgment device, such as a touch screen, keypad, mouse, or multiple mouse buttons, trackball, joystick, etc. Output device 704 can be an output device such as a display device of the projection effect judgment device. This display device can include, but is not limited to, liquid crystal displays (LCDs), light-emitting diode (LED) displays, and plasma displays. In some embodiments, the display device can be a touch screen.

[0183] The projection effect judging device of this application embodiment can be used to execute the technical solutions of the above-mentioned method embodiments of this application. Its implementation principle and technical effect are similar, and will not be repeated here.

[0184] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement any of the above-described projection effect judgment methods.

[0185] This invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the projection effect judgment method as described above.

[0186] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0187] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0188] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units.

Claims

1. A method of judging a projection effect, characterized by, The method comprises the following steps: acquiring projection data and image data of a projection area, wherein the projection data is obtained by projecting point cloud data on the projection area, and the image data is image data taken by the projection area; performing gray processing on the projection data and the image data to obtain an image gray image and a projection gray image; performing pixel filling processing on the projection gray image to obtain a dense projection gray image, wherein the kernel size of the filling processing is dynamically adjusted according to the row position in the projection image, a larger kernel size is used for rows closer to the bottom of the image, and a smaller kernel size is used for rows closer to the top of the image; subtracting the dense projection gray image from the image gray image to obtain a difference gray image; inputting the image gray image, the projection gray image, the dense projection gray image and the difference gray image into a preset neural network to obtain a projection label corresponding to the projection data, wherein the preset neural network is trained by a plurality of image gray image samples, a plurality of projection gray image samples, a plurality of dense projection gray image samples, a plurality of difference gray image samples and a plurality of preset projection labels, and the preset projection label comprises a good label, an upper bias label and a lower bias label; judging the projection effect according to the projection label.

2. The method of claim 1, wherein, The method of judging the projection effect according to the projection label comprises the following steps: if the projection label corresponding to the projection data is the good label, it is judged that the projection effect is accurate; if the projection label corresponding to the projection data is the upper bias label, it is judged that the projection effect is inaccurate and the projection is upwardly biased; if the projection label corresponding to the projection data is the lower bias label, it is judged that the projection effect is inaccurate and the projection is downwardly biased.

3. The method of claim 1, wherein, After judging the projection effect according to the projection label, the method further comprises the following step: adjusting the projection according to the projection effect.

4. A projection effect judging apparatus, characterized by comprising: The method comprises the following steps: at least one processor and a memory; the memory stores computer execution instructions; the at least one processor executes the computer execution instructions stored in the memory, so that the at least one processor executes the projection effect judging method according to any one of claims 1 to 3.

5. A computer readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and when the processor executes the computer execution instructions, the projection effect judging method according to any one of claims 1 to 3 is realized.

6. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to realize the projection effect judging method according to any one of claims 1 to 3.

Images