Autonomous driving simulation test method, device, system, apparatus and medium

Abstract

The present disclosure provides an automatic driving simulation test method, device, system, equipment and medium, relates to the technical field of artificial intelligence, in particular to the technical field of intelligent traffic and automatic driving. The scheme is: in the case that the ISP receives the original image sent by the processor, the ISP sends a data query request to the processor; the processor sends target associated data associated with the first identification information to the ISP; the ISP performs first processing on the original image based on the target associated data to obtain a target image, and sends the target image to an automatic driving simulation test platform; the automatic driving simulation test platform uses the target image to perform simulation test on the vehicle, thereby effectively realizing simulation of the vehicle-mounted camera sensor, providing high-quality image processing results matched in a real shooting scene, and performing simulation test on the vehicle based on the image processing results, thereby improving the authenticity and effectiveness of the vehicle automatic driving simulation test.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically to the fields of intelligent transportation and autonomous driving technology, and in particular to autonomous driving simulation testing methods, devices, systems, equipment, and media. Background Technology

[0002] With the development of autonomous driving technology, the autonomous driving functions of vehicles are becoming increasingly mature. Vehicles operating in autonomous driving mode (e.g., driverless) can free drivers from some driving-related responsibilities. To ensure sufficient safety and reliability of autonomous vehicles, extensive simulation testing is necessary. Therefore, how to implement autonomous driving simulation testing is crucial. Summary of the Invention

[0003] This disclosure provides a method, apparatus, system, device, and medium for autonomous driving simulation testing.

[0004] According to one aspect of this disclosure, an autonomous driving simulation testing method is provided, comprising: upon receiving an original image sent by a processor through an image signal processing (ISP), sending a data query request to the processor via the ISP; wherein the data query request includes first identification information of the original image; sending target association data associated with the first identification information to the ISP via the processor, wherein the target association data is obtained by the processor in response to the data query request, and the target association data is used to indicate device information of a target vehicle-mounted camera that acquired the original image and shooting information of the target vehicle-mounted camera when acquiring the original image; performing a first processing on the original image based on the target association data by the ISP to obtain a target image, and sending the target image to an autonomous driving simulation testing platform; and conducting a simulation test on a vehicle using the target image by the autonomous driving simulation testing platform.

[0005] According to another aspect of this disclosure, an autonomous driving simulation testing device is provided, comprising: a first sending module, configured to send a data query request to the processor via the ISP when the ISP receives an original image sent by the processor; wherein the data query request includes first identification information of the original image; a second sending module, configured to send target association data associated with the first identification information to the ISP via the processor, wherein the target association data is obtained by the processor in response to the data query request, and the target association data is used to indicate device information of a target vehicle-mounted camera that acquires the original image and shooting information of the target vehicle-mounted camera when acquiring the original image; a third sending module, configured to perform a first processing on the original image based on the target association data via the ISP to obtain a target image, and send the target image to an autonomous driving simulation testing platform; and a testing module, configured to perform simulation testing on a vehicle using the target image via the autonomous driving simulation testing platform.

[0006] According to another aspect of this disclosure, an autonomous driving simulation testing system is provided, comprising: an image signal processing (ISP), a processor, and an autonomous driving simulation testing platform, wherein the ISP is configured to send a data query request to the processor upon receiving an original image sent by the processor; wherein the data query request includes first identification information of the original image; the processor is configured to send target association data associated with the first identification information to the ISP, wherein the target association data is obtained by the processor in response to the data query request, and the target association data is used to indicate device information of a target vehicle-mounted camera sensor that acquired the original image and shooting information of the target vehicle-mounted camera sensor when acquiring the original image; the ISP is further configured to perform a first processing on the original image based on the target association data to obtain a target image, and send the target image to the autonomous driving simulation testing platform; the autonomous driving simulation testing platform is configured to perform simulation testing on a vehicle using the target image.

[0007] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the autonomous driving simulation test method described in the first aspect embodiment of this disclosure.

[0008] According to another aspect of this disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the autonomous driving simulation test method described in the first aspect of this disclosure.

[0009] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the autonomous driving simulation testing method described in the first aspect of this disclosure.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0012] Figure 1 This is a flowchart illustrating the autonomous driving simulation testing method provided in Embodiment 1 of this disclosure;

[0013] Figure 2 This is a flowchart illustrating the autonomous driving simulation testing method provided in Embodiment 2 of this disclosure;

[0014] Figure 3 This is a flowchart illustrating the autonomous driving simulation testing method provided in Embodiment 3 of this disclosure;

[0015] Figure 4 This is a flowchart illustrating the autonomous driving simulation testing method provided in Embodiment 4 of this disclosure;

[0016] Figure 5 This is a schematic diagram of the structure of the autonomous driving simulation test device provided in Embodiment 5 of this disclosure;

[0017] Figure 6 This is a schematic diagram of the structure of the autonomous driving simulation test system provided in Embodiment Six of this disclosure;

[0018] Figure 7 This is a schematic diagram of the structure of the autonomous driving simulation test system provided in the embodiments of this disclosure;

[0019] Figure 8 This is a schematic diagram of the FPGA structure provided in the embodiments of this disclosure;

[0020] Figure 9 This is a block diagram of an electronic device used to implement the autonomous driving simulation test method of the embodiments of this disclosure. Detailed Implementation

[0021] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0022] With the rapid development of autonomous driving technology, sensor technology plays a crucial role in the development of autonomous driving systems and the improvement of safety performance. As one of the core sensors in autonomous driving, onboard camera sensors help autonomous vehicles collect image information about their surroundings, enabling early warning and recognition functions. Simulation technology for onboard camera sensors can reduce system simulation costs, solve the problem of low density of typical scenarios encountered in autonomous driving simulation testing, form an efficient data testing closed loop, improve the flexibility and scalability of autonomous driving simulation testing, and provide measurable data for autonomous driving simulation testing.

[0023] In related technologies, the simulation of automotive camera sensors requires full physical-level modeling. This necessitates obtaining the underlying parameters of automotive camera sensor components (such as Complementary Metal Oxide Semiconductor (CMOS) chips and ISPs) and a deep understanding of the underlying hardware knowledge of automotive camera sensors. This is essentially something that can only be done by automotive camera sensor manufacturers or even chip suppliers. Autonomous driving companies find it difficult to achieve true camera sensor simulation. In hardware-in-the-loop (HIL) simulation testing, testing is typically done by connecting physical automotive camera sensors. This allows testing both perception algorithms and the sensor's functionality and performance. However, this approach has the drawback of being unable to simulate image data from real-world autonomous driving scenarios offline, resulting in high costs and poor flexibility.

[0024] Therefore, in response to the above problems, this disclosure proposes an autonomous driving simulation testing method, apparatus, system, equipment, and medium.

[0025] The following description, with reference to the accompanying drawings, details an autonomous driving simulation testing method, apparatus, system, device, and medium according to embodiments of the present disclosure.

[0026] Figure 1 This is a flowchart illustrating the autonomous driving simulation testing method provided in Embodiment 1 of this disclosure.

[0027] This disclosure illustrates an example of an autonomous driving simulation test method configured in an autonomous driving simulation test device. This autonomous driving simulation test device can be applied to any electronic device so that the electronic device can perform autonomous driving simulation test functions.

[0028] Among them, electronic devices can be any device with computing capabilities, such as personal computers (PCs), mobile terminals, servers, etc. Mobile terminals can be hardware devices with various operating systems, touch screens and / or displays, such as in-vehicle devices, mobile phones, tablets, personal digital assistants, wearable devices, etc.

[0029] like Figure 1 As shown, the autonomous driving simulation testing method may include the following steps:

[0030] Step 101: When the image signal processing receives the raw image sent by the processor, a data query request is sent to the processor via the ISP.

[0031] The data query request includes the first identification information of the original image.

[0032] As an example, the processor may receive an image acquisition request and, in response to the request, send the raw image to the Image Signal Processor (ISP), meaning the processor sends the image to the ISP in a controlled manner.

[0033] As another example, after the processor obtains the raw image directly, it sends the raw image to the ISP.

[0034] In an exemplary embodiment, the processor can not only respond to ISP commands but also transmit images. Therefore, as an example, the processor can be a Field Programmable Gate Array (FPGA).

[0035] Furthermore, when the ISP receives the original image sent by the processor, the ISP sends a data query request to the processor. The data query request may include the first identification information of the original image. The first identification information may be information that uniquely identifies the original image, such as the image identifier (ID).

[0036] Step 102: Send the target association data associated with the first identification information to the ISP through the processor.

[0037] Among them, the target association data is obtained by the processor in response to the data query request. The target association data is used to indicate the device information of the target vehicle camera sensor that acquires the original image and the shooting information of the target vehicle camera sensor when acquiring the original image.

[0038] To simulate an onboard camera sensor, in an exemplary embodiment, the processor, in response to a data query request, retrieves stored target association data associated with the first identification information of the original image from the memory. The target association data is used to indicate the device information of the target onboard camera sensor that uses the original image and the shooting information when the target onboard camera sensor acquires the original image. For example, the target association data includes the model, size, effective pixels, and exposure time of the target onboard camera sensor when acquiring the original image.

[0039] It is important to understand that in related technologies, because the processor (e.g., FPGA) does not simulate I2C response functionality, meaning the processor does not respond to the ISP's command set, the ISP's firmware can only be flashed with special firmware to disable all image processing functions. All image data input to the ISP will be directly transmitted to the subsequent autonomous driving simulation test platform. Furthermore, the images simulated by the processor from the vehicle's camera are not the original images, but rather YUV / RGB images pre-processed by the ISP. This is inconsistent with the ISP's normal operating mode. Therefore, in an exemplary embodiment, if the processor cannot respond to the data query request, i.e., the ISP cannot obtain the processor's status response, the ISP enters a failed safety mode and does not send the original image to the autonomous driving simulation test platform.

[0040] Step 103: The original image is processed by the ISP based on the target association data to obtain the target image, and the target image is sent to the autonomous driving simulation test platform.

[0041] Furthermore, the target image can be obtained by first processing the original image with target-related data through the ISP, and then the target image can be sent to the autonomous driving simulation test platform through the ISP.

[0042] Step 104: Simulate the vehicle using the target image through the autonomous driving simulation test platform.

[0043] Furthermore, the vehicle can be simulated and tested using target images through an autonomous driving simulation testing platform.

[0044] In summary, by receiving the original image from the processor via the image signal processing ISP, the ISP sends a data query request to the processor; the processor sends target association data associated with the first identification information to the ISP; the ISP performs a first processing on the original image based on the target association data to obtain the target image, and sends the target image to the autonomous driving simulation test platform; the autonomous driving simulation test platform uses the target image to conduct simulation tests on the vehicle. Thus, in response to the data query request sent by the ISP, the processor sends the target association data associated with the first identification information of the original image to the ISP. Therefore, the ISP performs a first processing on the original image based on the target association data to obtain the target image. This effectively realizes the simulation of the onboard camera sensor, providing high-quality image processing results that match real-world shooting scenarios. Based on these image processing results, vehicle simulation tests are conducted, improving the realism and effectiveness of autonomous driving simulation tests.

[0045] To clearly illustrate how the above embodiments send target association data associated with the identification information of the original image to the ISP via the processor, this disclosure proposes another autonomous driving simulation test method.

[0046] Figure 2 This is a flowchart illustrating the autonomous driving simulation testing method provided in Embodiment 2 of this disclosure.

[0047] like Figure 2 As shown, the autonomous driving simulation testing method may include the following steps:

[0048] Step 201: After the ISP receives the raw image sent by the processor, it sends a data query request to the processor through the ISP.

[0049] The data query request includes the first identification information of the original image.

[0050] Step 202: In response to the data query request, the processor determines the target associated data stored in the stored data that is associated with the first identification information, based on the first identification information.

[0051] Among them, the target association data is obtained by the processor in response to the data query request. The target association data is used to indicate the device information of the target vehicle camera sensor that acquires the original image and the shooting information of the target vehicle camera sensor when acquiring the original image.

[0052] It should be understood that the stored data in the memory may include the original images captured by the vehicle-mounted camera sensor, as well as the device information of the vehicle-mounted sensor corresponding to each original image and the shooting information when the original image was captured. Therefore, in an exemplary embodiment, the processor can respond to a data query request and determine the target associated data stored in association with the first identification information from the stored data according to the first identification information.

[0053] Step 203: The processor sends the target associated data to the ISP via the serial bus I2C communication protocol.

[0054] To improve transmission efficiency, in an exemplary embodiment, the processor can send the original image to the ISP via a high-speed image transmission channel, and the processor can send the target-related data to the ISP via the Inter-Integrated Circuit (I2C) communication protocol. In other words, the target-related data is sent to the ISP via the I2C channel.

[0055] Step 204: The original image is processed by the ISP based on the target association data to obtain the target image, and the target image is sent to the autonomous driving simulation test platform.

[0056] Step 205: Simulate the vehicle using the target image through an autonomous driving simulation test platform.

[0057] In summary, by responding to a data query request, the processor determines the target associated data stored in the storage data based on the first identification information. The processor then sends the target associated data to the ISP via the serial bus I2C communication protocol. This effectively simulates the processor's response to the ISP's query for the target associated data. Based on the query results of the target associated data, autonomous driving simulation tests of the vehicle are conducted, improving the realism and effectiveness of the autonomous driving simulation tests.

[0058] To clearly illustrate how the above embodiments perform a first processing on the original image based on the target association data through the ISP to obtain the target image and send the target image to the autonomous driving simulation test platform, this disclosure proposes another autonomous driving simulation test method.

[0059] Figure 3 This is a flowchart illustrating the autonomous driving simulation testing method provided in Embodiment 3 of this disclosure.

[0060] like Figure 3 As shown, the autonomous driving simulation testing method may include the following steps:

[0061] Step 301: After the ISP receives the raw image sent by the processor, it sends a data query request to the processor through the ISP.

[0062] The data query request includes the first identification information of the original image.

[0063] Step 302: The processor sends the target association data associated with the first identification information to the ISP.

[0064] Among them, the target association data is obtained by the processor in response to the data query request. The target association data is used to indicate the device information of the target vehicle camera sensor that acquires the original image and the shooting information of the target vehicle camera sensor when acquiring the original image.

[0065] Step 303: Obtain a second image processing algorithm that matches the target-related data from the first image processing algorithm that has passed the functional test through the ISP.

[0066] To improve the accuracy of the ISP in processing the original image, in an exemplary embodiment, the image processing algorithm in the ISP can be functionally tested, and the image processing algorithm that fails the functional test can be debugged until the image processing algorithm passes the functional test. Thus, the image processing algorithm that has passed the functional test is used to process the original image, thereby improving the accuracy of image processing. As a result, the ISP can provide the autonomous driving simulation test platform with high-quality image processing results that match the real shooting scene.

[0067] Therefore, as an example, the processor acquires multiple test images collected by the vehicle's onboard camera sensor under different driving environments and test association data associated with each test image; for any one of the multiple test images, a first image processing algorithm in the ISP that matches the test association data associated with any one test image is used to process the test image; when the difference between the processed test image and the expected processing result of any one test image is less than or equal to a set threshold, the first image processing algorithm for processing any one test image is determined to have passed the functional test; when the difference between the processed test image and the expected processing result of any one test image is greater than the set threshold, the first image processing algorithm for processing any one test image is determined to have failed the functional test.

[0068] In other words, different test-related data correspond to different first image processing algorithms. The ISP can use the first image processing algorithm that matches the test-related data to process the corresponding test image. If the difference between the processed test image and the expected processing result of any test image is less than or equal to a set threshold, it means that the first image processing algorithm for processing the test image has passed the functional test; otherwise, it has failed the functional test.

[0069] To improve the processing performance of the ISP, furthermore, if the first image processing algorithm for processing any test image fails the functional test, the parameters of the first image processing algorithm for processing any test image are adjusted at least once until the first image processing algorithm after at least one adjustment passes the functional test.

[0070] Therefore, the ISP can obtain a second image processing algorithm that matches the target-related data from the first image processing algorithm that has passed the functional test. That is, it obtains the image processing algorithm that matches the target-related data and has passed the functional test.

[0071] Step 304: The original image is processed by the second image processing algorithm through the ISP to obtain the target image.

[0072] It's important to note that the image quality captured by the vehicle's camera sensors during driving is affected by changes in lighting conditions, such as weather changes and entering / exiting tunnels. For example, in traffic light recognition, the image quality changes due to ambient lighting conditions, causing the red light to appear yellowish. This can lead to the perception network incorrectly identifying the light as yellow, resulting in misjudgment and vehicle intervention. This is because on sunny days, the light is good, the exposure time for the traffic light area is very short, and the gain is low, making the red light appear very red. On cloudy days, the light is dim, the exposure time is longer, and the gain is higher, causing the red light to appear yellowish.

[0073] Therefore, by employing a second image processing algorithm in the ISP to perform a first processing on the original image to obtain the target image, that is, by using an image processing algorithm that has passed functional testing to process the original image, the accuracy of the ISP in processing the original image is improved. For example, adjusting parameters such as hue and color saturation, and adjusting gamma and image brightness can improve the image quality of red lights, ensuring the correct recognition of traffic lights under different lighting conditions. This allows the ISP to provide the autonomous driving simulation test platform with high-quality image processing results that match the actual shooting scene. The first processing may include, but is not limited to, lens shading correction, noise removal, denoising, interpolation, and white balance.

[0074] Step 305: Send the target image to the autonomous driving simulation test platform via ISP.

[0075] Step 306: Simulate the vehicle using the target image through an autonomous driving simulation test platform.

[0076] Therefore, the ISP obtains a second image processing algorithm that matches the target-related data from the first image processing algorithm that has passed the functional test; the ISP then uses the second image processing algorithm to perform first processing on the original image to obtain the target image. Thus, by using the image processing algorithm that passed the functional test to process the original image, the accuracy of image processing is improved, thereby enabling the ISP to provide the autonomous driving simulation test platform with high-quality image processing results that match the real shooting scenario.

[0077] In an exemplary embodiment, the processor can send the raw image to the ISP in a controlled manner. Therefore, when the ISP receives the raw image sent by the processor, before sending a data query request to the processor through the ISP, an image acquisition request can be sent to the processor through the ISP. In response to the image acquisition request, the processor sends the raw image to the ISP. The following describes the process in conjunction with... Figure 4 Please provide a detailed explanation.

[0078] Figure 4 This is a flowchart illustrating the autonomous driving simulation testing method provided in Embodiment 4 of this disclosure.

[0079] like Figure 4 As shown, the autonomous driving simulation testing method may include the following steps:

[0080] Step 401: Send an image acquisition request to the processor via the ISP.

[0081] The image acquisition request includes the ISP's second identification information.

[0082] In an exemplary embodiment, in response to the ISP receiving a trigger instruction from the autonomous driving simulation test platform, it may send an image acquisition request to the processor, wherein the image acquisition request may include the second identification information of the ISP.

[0083] It's important to understand that there can be a mapping relationship between the vehicle-mounted camera sensors and the ISP (In-Service Provider). This mapping relationship can be stored in the processor. For example, if there are multiple vehicle-mounted camera sensors and multiple ISPs, each sensor can be paired with a different ISP. The processor connects to the corresponding vehicle-mounted camera sensor via different pins and sends the corresponding sensor data to the ISP at the corresponding communication address. In other words, different ISPs process the raw images acquired by different vehicle-mounted camera sensors, and the raw images acquired by each sensor are stored in the same image queue.

[0084] Step 402: Send the raw image to the ISP via the processor.

[0085] The original image is the vehicle camera image that the processor determines from the vehicle camera images collected by multiple vehicle camera sensors stored in the memory, based on the second identification information, and that matches the second identification information.

[0086] Furthermore, the processor can query the mapping relationship between the vehicle-mounted camera sensor and the ISP based on the second identification information, and obtain the target vehicle-mounted camera sensor that matches the second identification information from multiple vehicle-mounted camera sensors. The image acquired by the target vehicle-mounted camera sensor that matches the second identification information and is already stored in the memory is used as the original image. It should be noted that the memory can use a queue storage method to store the vehicle-mounted camera images acquired by each vehicle-mounted camera sensor. Therefore, the first image in the image queue corresponding to the target vehicle-mounted camera sensor that matches the second identification information can be used as the original image. It should also be noted that the vehicle-mounted camera sensor can acquire vehicle-mounted camera images under different driving environments. The vehicle-mounted camera sensor can send these images to the memory for storage. Thus, by using pre-collected image data from real-world autonomous driving scenarios, and then reading this image data from the memory during autonomous driving simulation testing, offline simulation of real-world autonomous driving scenarios is achieved, which is low-cost and highly flexible.

[0087] As an example, the processor sends the original image to the camera output interface based on the camera interface protocol; the original image is subjected to a second processing through the camera output interface to obtain a second-processed original image, and the second-processed original image is sent to the camera input interface; wherein, the second processing includes at least one of the following processes: data length detection processing, frequency conversion processing between the receiving frequency of the original image received by the camera output interface and the sending frequency of the original image sent by the camera input interface; the second-processed original image is sent to the ISP through the camera input interface.

[0088] In other words, to improve the transmission efficiency of the original image, in the exemplary embodiment, the processor sends the original image to the camera output interface at high speed. To enable the ISP to effectively process the original data, the camera output interface, after receiving the original image, can perform a second processing on the original image to obtain a second-processed original image, and then send the second-processed original image to the camera input interface; the second-processed original image is then sent to the ISP through the camera input interface. The second processing may include, but is not limited to, data length detection processing, frequency conversion processing between the receiving frequency of the original image received by the camera output interface and the sending frequency of the original image sent by the camera input interface, and level conversion, etc.

[0089] Step 403: After the ISP receives the raw image sent by the processor, it sends a data query request to the processor through the ISP.

[0090] The data query request includes the first identification information of the original image.

[0091] Step 404: The processor sends the target association data associated with the first identification information to the ISP.

[0092] Among them, the target association data is obtained by the processor in response to the data query request. The target association data is used to indicate the device information of the target vehicle camera sensor that acquires the original image and the shooting information of the target vehicle camera sensor when acquiring the original image.

[0093] Step 405: The original image is processed by the ISP based on the target association data to obtain the target image, and the target image is sent to the autonomous driving simulation test platform.

[0094] Step 406: Simulate the vehicle using the target image through the autonomous driving simulation test platform.

[0095] In summary, the ISP sends an image acquisition request to the processor; the processor then sends the raw image to the ISP. The processor then sends the actually acquired camera sensor data to the ISP in a controlled manner according to the camera interface protocol. The ISP then performs initial processing on the raw image based on the target-related data to obtain the target image. This effectively simulates the vehicle-mounted camera sensor, providing high-quality image processing results that match real-world shooting scenarios. Based on these image processing results, vehicle simulation tests are conducted, improving the realism and effectiveness of autonomous driving simulation tests.

[0096] To achieve the above embodiments, this disclosure proposes an autonomous driving simulation test device.

[0097] Figure 5 This is a schematic diagram of the structure of the autonomous driving simulation test device provided in Embodiment 5 of this disclosure.

[0098] like Figure 5 As shown, the autonomous driving simulation test device 500 includes: a first sending module 510, a second sending module 520, a third sending module 530, and a test module 540.

[0099] The system includes a first sending module 510, which sends a data query request to the processor via the ISP when the ISP receives the original image sent by the processor; wherein the data query request includes first identification information of the original image; a second sending module 520, which sends target association data associated with the first identification information to the ISP via the processor; wherein the target association data is obtained by the processor in response to the data query request, and the target association data is used to indicate the device information of the target vehicle camera that acquired the original image and the shooting information of the target vehicle camera when acquiring the original image; a third sending module 530, which performs a first processing on the original image based on the target association data via the ISP to obtain a target image, and sends the target image to the autonomous driving simulation test platform; and a test module 540, which performs simulation testing on the vehicle using the target image via the autonomous driving simulation test platform.

[0100] As one possible implementation of this disclosure, the second sending module 520 is used to respond to a data query request by the processor, determine the target associated data stored in the stored data according to the first identification information, and send the target associated data to the ISP by the processor based on the I2C communication protocol.

[0101] As one possible implementation of this disclosure, the autonomous driving simulation test device 500 further includes: a first acquisition module, a first processing module, a first determination module, and a second determination module.

[0102] The system comprises: a first acquisition module, configured to acquire multiple test images collected by the vehicle's onboard camera sensor under different driving environments, and test association data associated with each test image; a first processing module, configured to process any one of the multiple test images using a first image processing algorithm in the ISP that matches the test association data associated with that test image; a first determination module, configured to determine that the first image processing algorithm for processing any test image passes the functional test when the difference between the processed test image and the expected processing result of any test image is less than or equal to a set threshold; and a second determination module, configured to determine that the first image processing algorithm for processing any test image fails the functional test when the difference between the processed test image and the expected processing result of any test image is greater than a set threshold.

[0103] As one possible implementation of this disclosure, the autonomous driving simulation test device 500 further includes an adjustment module.

[0104] The adjustment module is used to adjust the parameters of the first image processing algorithm for processing any test image at least once if the first image processing algorithm for processing any test image fails the functional test, until the first image processing algorithm after at least one adjustment passes the functional test.

[0105] As one possible implementation of this disclosure, the third sending module is configured to: obtain a second image processing algorithm that matches the target-related data from a first image processing algorithm that has passed the functional test via an ISP; perform a first processing on the original image using the second image processing algorithm via the ISP to obtain a target image; and send the target image to an autonomous driving simulation test platform via the ISP.

[0106] As one possible implementation of this disclosure, the autonomous driving simulation test device 500 further includes: a second acquisition module and a fourth transmission module.

[0107] The second acquisition module is used to send an image acquisition request to the processor through the ISP; wherein the image acquisition request includes the second identification information of the ISP; the fourth sending module is used to send a raw image to the ISP through the processor, wherein the raw image is a target vehicle camera sensor that matches the second identification information, determined by the processor from multiple vehicle camera images collected by vehicle camera sensors stored in the memory, based on the second identification information.

[0108] As one possible implementation of this disclosure, the fourth sending module is configured to send the original image to the camera output interface via the processor based on the camera interface protocol; perform a second processing on the original image via the camera output interface to obtain a second-processed original image, and send the second-processed original image to the camera input interface; wherein the second processing includes at least one of the following processes: data length detection processing, frequency conversion processing between the receiving frequency of the original image received by the camera output interface and the sending frequency of the original image sent by the camera input interface; and send the second-processed original image to the ISP via the camera input interface.

[0109] As one possible implementation of this disclosure, the autonomous driving simulation test device 500 further includes: a data acquisition module and a storage module.

[0110] The acquisition module is used to acquire vehicle-mounted camera images under different driving environments through the vehicle-mounted camera sensor; the storage module is used to send the vehicle-mounted camera images to the memory through the vehicle-mounted camera sensor to store the vehicle-mounted camera images.

[0111] The autonomous driving simulation testing apparatus of this disclosure, upon receiving an original image from a processor via an image signal processing ISP, sends a data query request to the processor through the ISP; the processor sends target association data associated with first identification information to the ISP; the ISP performs a first processing on the original image based on the target association data to obtain a target image, and sends the target image to an autonomous driving simulation testing platform; the autonomous driving simulation testing platform uses the target image to conduct simulation testing on a vehicle. Thus, in response to the data query request sent by the ISP, the processor sends target association data associated with the first identification information of the original image to the ISP, thereby the ISP performs a first processing on the original image based on the target association data to obtain the target image. This effectively realizes the simulation of the vehicle-mounted camera sensor, providing high-quality image processing results that match real-world shooting scenarios. Based on these image processing results, vehicle simulation testing is conducted, improving the realism and effectiveness of autonomous driving simulation testing.

[0112] To implement the above embodiments, this disclosure proposes an autonomous driving simulation testing system.

[0113] Figure 6 This is a schematic diagram of the structure of the autonomous driving simulation test system provided in Embodiment Six of this disclosure.

[0114] like Figure 6 As shown, the autonomous driving simulation test system 600 includes: ISP 610, processor 620 and autonomous driving simulation test platform 630.

[0115] The ISP 610 is configured to send a data query request to the processor upon receiving the original image from the processor; the data query request includes first identification information of the original image; the processor 620 is configured to send target association data associated with the first identification information to the ISP, wherein the target association data is obtained by the processor in response to the data query request, and the target association data is used to indicate the device information of the target vehicle-mounted camera sensor that acquired the original image and the shooting information of the target vehicle-mounted camera sensor when acquiring the original image; the ISP 610 is also configured to perform a first processing on the original image based on the target association data to obtain a target image, and send the target image to the autonomous driving simulation test platform 630; the autonomous driving simulation test platform 630 is configured to conduct simulation tests on the vehicle using the target image.

[0116] Based on any embodiment of this disclosure, such as Figure 7As shown, the autonomous driving simulation test system of this disclosure embodiment may include three parts: the first part is an online acquisition box, used to acquire and store camera sensor data (raw images) in a real autonomous driving scenario; the second part is an offline camera fax machine simulation system, used to receive pre-acquired camera sensor data, verify and package it, and transmit it to the camera connection cable through the camera output interface; the third part is an autonomous driving hardware system, which receives the camera sensor data sent by the offline camera fax machine simulation system from the camera hardware port through the camera input interface, and completes the simulation test according to the scenario of connecting real camera sensor data.

[0117] The online data acquisition box includes a camera sensor (vehicle-mounted camera sensor) 701, a data acquisition control unit 702, and a storage unit (memory) 703. The camera sensor 701 is a real vehicle-mounted camera sensor used to acquire image data in real autonomous driving scenarios. The data acquisition control unit 702 is used to control the camera sensor, verify data, and receive data. The storage unit 703 is used to store the scene data acquired by the camera sensor.

[0118] The offline camera sensor simulation system includes an FPGA 704 and a camera output interface 705. The FPGA 704 receives stored camera sensor data and transmits it according to the Camera Interface Protocol (MIPI). The camera output interface 705 is used for camera sensor data format and level conversion and transmits the data to the camera connection cable.

[0119] The autonomous driving hardware system includes a camera input interface 706, an image signal processing (ISP) 707, and an autonomous driving control unit (autonomous driving simulation test platform) 708.

[0120] The system includes: Camera Input Interface 706: Receives camera sensor data from the camera hardware port and transmits it to the subsequent image processing unit; Image Signal Processing (ISP) 707: Performs post-processing on the signal output from the camera sensor, converting the raw signal (raw image) acquired by the camera sensor (e.g., CMOS) into a color / black and white image acceptable to the human eye. The I2C controller in the ISP needs to read relevant information from the camera sensor (target-related data); otherwise, the ISP will enter a failure safety mode; Autonomous Driving Control Unit 708: The test object of the simulation system, used to receive and process data from different sensors to implement various safety and automated driving functions.

[0121] As one possible implementation of this disclosure, such as Figure 8 As shown, the FPGA may include a data receiving module 801, a data packing module 802, a data sending module 803, a control module 804, and an I2C communication response module 805.

[0122] The data receiving module 801 is used to receive pre-collected camera sensor data and perform data length and format detection. The data packaging module 802 is used to perform camera sensor data bit width and clock domain conversion, and then package the data and send it to the data sending module. The data sending module 803 is used to send camera sensor data to the camera output interface. The control module 804 is used to communicate with the host computer and complete the configuration and status reporting of each sub-module. The I2C communication response module 805 is used to implement the I2C communication protocol with the image signal processing (ISP) chip, return the correct parameter values ​​for the ISP's I2C command query, and ensure that the ISP enters the normal working state.

[0123] The autonomous driving simulation test system of this disclosure, upon receiving an original image from a processor via an image signal processing ISP, sends a data query request to the processor through the ISP; the processor sends target association data associated with first identification information to the ISP; the ISP performs a first processing on the original image based on the target association data to obtain a target image, and sends the target image to the autonomous driving simulation test platform; the autonomous driving simulation test platform uses the target image to conduct simulation tests on the vehicle. Thus, in response to the data query request sent by the ISP, the processor sends target association data associated with the first identification information of the original image to the ISP, thereby the ISP performs a first processing on the original image based on the target association data to obtain the target image. This effectively realizes the simulation of the vehicle-mounted camera sensor, provides high-quality image processing results that match real-world shooting scenarios, and improves the realism and effectiveness of autonomous driving simulation tests by conducting simulation tests on the vehicle based on these image processing results.

[0124] It should be noted that the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the technical solution disclosed herein are all carried out with the consent of the user, and all comply with the provisions of relevant laws and regulations, and do not violate public order and good morals.

[0125] To implement the above embodiments, this disclosure also provides an electronic device, which may include at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to execute the autonomous driving simulation test method proposed in any of the above embodiments of this disclosure.

[0126] To implement the above embodiments, this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the autonomous driving simulation test method proposed in any of the above embodiments of this disclosure.

[0127] To implement the above embodiments, this disclosure also provides a computer program product, which includes a computer program that, when executed by a processor, implements the autonomous driving simulation testing method proposed in any of the above embodiments of this disclosure.

[0128] Figure 9 A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0129] like Figure 9 As shown, device 900 includes a computing unit 901, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 902 or a computer program loaded from storage unit 908 into random access memory (RAM) 903. RAM 903 may also store various programs and data required for the operation of device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0130] Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of monitors, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0131] The computing unit 901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as an autonomous driving simulation testing method. For example, in some embodiments, the autonomous driving simulation testing method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the sample distorted document image generation method described above can be performed. Alternatively, in other embodiments, the computing unit 901 can be configured to perform the autonomous driving simulation testing method by any other suitable means (e.g., by means of firmware).

[0132] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0133] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0134] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0135] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0136] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0137] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0138] It's important to note that artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0139] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0140] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An autonomous driving simulation testing method, comprising: When the image signal processing ISP receives the raw image sent by the processor, the ISP sends a data query request to the processor; wherein the data query request includes the first identification information of the raw image; In response to the data query request, the processor determines the target associated data stored in the stored data according to the first identification information, and sends the target associated data to the ISP via the processor based on the serial bus I2C communication protocol; wherein, the target associated data is used to indicate the device information of the target vehicle camera sensor that acquired the original image and the shooting information of the target vehicle camera sensor when acquiring the original image; The ISP performs a first processing on the original image based on the target association data to obtain the target image, and then sends the target image to the autonomous driving simulation test platform. The autonomous driving simulation test platform uses the target image to conduct simulation tests on the vehicle.

2. The method according to claim 1, wherein, Before the first processing of the original image based on the target association data by the ISP to obtain the target image, and before sending the target image to the autonomous driving simulation test platform, the process includes: The processor acquires multiple test images collected by the vehicle's onboard camera sensor under different driving environments, as well as test association data associated with each test image. For any one of the plurality of test images, a first image processing algorithm that matches the test association data associated with the test image in the ISP is used to process the test image. If the difference between the processed test image and the expected processing result of any test image is less than or equal to a set threshold, the first image processing algorithm for processing any test image is determined to have passed the functional test. If the difference between the processed test image and the expected processing result of any test image is greater than a set threshold, it is determined that the first image processing algorithm for processing any test image has failed the functional test.

3. The method according to claim 2, wherein, The method further includes: If the first image processing algorithm for processing any of the test images fails the functional test, the parameters of the first image processing algorithm for processing any of the test images shall be adjusted at least once until the first image processing algorithm after at least one adjustment passes the functional test.

4. The method according to claim 3, wherein, The step of performing a first processing on the original image based on the target association data through the ISP to obtain a target image, and sending the target image to the autonomous driving simulation test platform, includes: The ISP obtains a second image processing algorithm that matches the target-related data from the first image processing algorithm that has passed the functional test. The original image is processed by the ISP using the second image processing algorithm to obtain the target image; The target image is sent to the autonomous driving simulation test platform via the ISP.

5. The method according to claim 1, wherein, Before the method involves the ISP sending a data query request to the processor upon receiving the raw image from the processor, the method further includes: The image acquisition request is sent to the processor through the ISP; wherein the image acquisition request includes the second identification information of the ISP; The processor sends the original image to the ISP, wherein the original image is a vehicle camera image that the processor determines from multiple vehicle camera images collected by vehicle camera sensors that match the second identification information, based on the second identification information.

6. The method according to claim 5, wherein, Sending the raw image to the ISP via the processor includes: The processor sends the raw image to the camera output interface based on the camera interface protocol. The original image is subjected to a second processing through the camera output interface to obtain a second-processed original image, and the second-processed original image is sent to the camera input interface; wherein, the second processing includes at least one of the following processes: data length detection processing, frequency conversion processing between the receiving frequency of the original image received by the camera output interface and the sending frequency of the original image sent by the camera input interface; The second processed raw image is sent to the ISP through the camera input interface.

7. The method according to claim 5, wherein, Before sending the original image to the ISP via the processor, the method further includes: The vehicle-mounted camera sensor acquires images of the vehicle under different driving conditions. The vehicle-mounted camera sensor sends the images from the vehicle-mounted camera to a memory for storage.

8. An autonomous driving simulation testing device, comprising: The first sending module is configured to send a data query request to the processor via the image signal processing ISP when the image signal processing ISP receives the original image sent by the processor; wherein the data query request includes first identification information of the original image; The second sending module is configured to, in response to the data query request by the processor, determine the target associated data stored in the stored data according to the first identification information, and send the target associated data to the ISP via the processor based on the serial bus I2C communication protocol; wherein, the target associated data is obtained by the processor in response to the data query request, and the target associated data is used to indicate the device information of the target vehicle camera that acquired the original image and the shooting information of the target vehicle camera when acquiring the original image; The third sending module is used to perform a first processing on the original image based on the target association data by the ISP to obtain the target image, and send the target image to the autonomous driving simulation test platform; The testing module is used to perform simulation tests on the vehicle using the target image through the autonomous driving simulation testing platform.

9. The apparatus according to claim 8, wherein, The device further includes: The first acquisition module is used to acquire multiple test images collected by the vehicle's on-board camera sensor under different driving environments, as well as test association data associated with each test image, through the processor. The first processing module is used to process any one of the plurality of test images by employing a first image processing algorithm that matches the test association data associated with the test image in the ISP. The first determining module is used to determine that the first image processing algorithm for processing any test image passes the functional test when the difference between the processed test image and the expected processing result of any test image is less than or equal to a set threshold. The second determining module is used to determine that the first image processing algorithm for processing any test image has failed the functional test when the difference between the processed test image and the expected processing result of any test image is greater than a set threshold.

10. The apparatus according to claim 9, wherein, The device further includes: An adjustment module is configured to adjust the parameters of the first image processing algorithm for processing any test image at least once if the first image processing algorithm for processing any test image fails the functional test, until the first image processing algorithm after at least one adjustment passes the functional test.

11. The apparatus according to claim 10, wherein, The third sending module is used for: The ISP obtains a second image processing algorithm that matches the target-related data from the first image processing algorithm that has passed the functional test. The original image is processed by the ISP using the second image processing algorithm to obtain the target image; The target image is sent to the autonomous driving simulation test platform via the ISP.

12. The apparatus according to claim 8, wherein, The device further includes: The second acquisition module is used to send an image acquisition request to the processor through the ISP; wherein the image acquisition request includes the second identification information of the ISP; The fourth sending module is used to send the original image to the ISP through the processor, wherein the original image is the vehicle camera image that the processor determines from the vehicle camera images collected by multiple vehicle camera sensors that match the second identification information stored in the memory, based on the second identification information.

13. The apparatus according to claim 12, wherein, The fourth sending module is used for: The processor sends the raw image to the camera output interface based on the camera interface protocol. The original image is subjected to a second processing through the camera output interface to obtain a second-processed original image, and the second-processed original image is sent to the camera input interface; wherein, the second processing includes at least one of the following processes: data length detection processing, frequency conversion processing between the receiving frequency of the original image received by the camera output interface and the sending frequency of the original image sent by the camera input interface; The second processed raw image is sent to the ISP through the camera input interface.

14. The apparatus according to claim 12, wherein, The device further includes: The acquisition module is used to acquire images of the vehicle under different driving environments through the vehicle-mounted camera sensor; The storage module is used to send the vehicle-mounted camera images to the memory via the vehicle-mounted camera sensor for storing the vehicle-mounted camera images.

15. An autonomous driving simulation testing system, comprising: Image signal processing ISP, processor, and autonomous driving simulation test platform, among which, The ISP is configured to send a data query request to the processor upon receiving the original image sent by the processor; wherein the data query request includes first identification information of the original image; The processor is configured to respond to the data query request, determine the target associated data stored in the stored data according to the first identification information, and send the target associated data to the ISP based on the serial bus I2C communication protocol; wherein the target associated data is used to indicate the device information of the target vehicle camera sensor that acquires the original image and the shooting information of the target vehicle camera sensor when acquiring the original image; The ISP is also used to perform a first processing on the original image based on the target association data to obtain a target image, and send the target image to the autonomous driving simulation test platform; The autonomous driving simulation test platform is used to conduct simulation tests on vehicles using the target image.

16. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

17. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.

18. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-7.

Images