Method, apparatus, electronic device, and storage medium for processing sensor information

Abstract

The present disclosure provides a method, apparatus, electronic device, and storage medium for processing sensor information, which relates to the field of artificial intelligence technology, and particularly relates to the fields of computer vision and autonomous driving. The specific implementation solution is as follows: Obtain the configuration information of the sensor; wherein, the configuration information of the sensor is used to determine the calibration parameters of the sensor; display an imaging simulation diagram of the sensor in a target scene, and the imaging simulation diagram includes an image of a calibration reference object; wherein, the imaging simulation diagram is generated based on the calibration parameters and the virtual scene data of the target scene, and the virtual scene data includes three-dimensional data of the calibration reference object. According to the embodiments of the present disclosure, the user can determine whether the perception effect of the sensor under the current configuration meets the calibration requirements based on the imaging simulation diagram and the calibration reference object therein, which has the advantages of comprehensive, intuitive, and efficient calibration.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and particularly to the fields of computer vision and autonomous driving. Background Art

[0002] Autonomous driving highly depends on various sensors (such as cameras, radars, etc.). The deployment position of the sensor is the key to restricting the sensing ability of the sensor. For example, when arranging a camera close to the vehicle surface, the vehicle surface will block the imaging of the camera, affecting the shooting range of the camera. Based on this, before confirming the deployment position of the sensor, it is necessary to check the sensing effect of the sensor. Summary of the Invention

[0003] The present disclosure provides a method, an apparatus, an electronic device, and a storage medium for processing sensor information.

[0004] According to one aspect of the present disclosure, there is provided a method for processing sensor information, including:

[0005] Obtaining configuration information of a sensor; wherein, the configuration information of the sensor is used to determine calibration parameters of the sensor;

[0006] Displaying an imaging simulation diagram of the sensor in a target scene, where the imaging simulation diagram includes an image of a calibration reference object; wherein, the imaging simulation diagram is generated based on the calibration parameters and virtual scene data of the target scene, and the virtual scene data includes three-dimensional data of the calibration reference object.

[0007] According to another aspect of the present disclosure, there is provided an apparatus for processing sensor information, including:

[0008] A configuration information acquisition module, configured to obtain configuration information of a sensor; wherein, the configuration information of the sensor is used to determine calibration parameters of the sensor;

[0009] An image display module, configured to display an imaging simulation diagram of the sensor in a target scene, where the imaging simulation diagram includes an image of a calibration reference object; wherein, the imaging simulation diagram is generated based on the calibration parameters and virtual scene data of the target scene, and the virtual scene data includes three-dimensional data of the calibration reference object.

[0010] According to another aspect of the present disclosure, there is provided an electronic device, including:

[0011] At least one processor; and

[0012] A memory communicatively connected to the at least one processor; wherein,

[0013] The memory stores instructions executable by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to execute the method of any embodiment in the present disclosure.

[0014] According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method of any embodiment in the present disclosure.

[0015] According to another aspect of the present disclosure, there is provided a computer program product including a computer program, which implements the method of any embodiment in the present disclosure when executed by a processor.

[0016] The technical solution of the embodiment of the present disclosure generates an imaging simulation diagram based on the acquired configuration information of the sensor, and presents the sensing area of the sensor in the target scene in the form of imaging simulation. Moreover, by setting the three-dimensional data of the calibration reference object in the virtual scene data of the target scene, the imaging simulation diagram includes the image of the calibration reference object. Based on this, the user can determine whether the sensing effect of the sensor under the current configuration meets the calibration requirements based on the imaging simulation diagram and the calibration reference object therein, which has the advantages of comprehensive, intuitive, and efficient calibration.

[0017] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. Description of the Drawings

[0018] The drawings are used to better understand the solution and do not constitute a limitation to the present disclosure. Among them:

[0019] Figure 1 is a schematic flowchart of a method for processing sensor information according to an embodiment of the present disclosure;

[0020] Figure 2 is a schematic diagram of an imaging simulation diagram according to an embodiment of the present disclosure;

[0021] Figure 3 is a schematic diagram of another imaging simulation diagram according to an embodiment of the present disclosure

[0022] Figure 4 is a schematic diagram of a three-dimensional scene model corresponding to virtual scene data according to an embodiment of the present disclosure;

[0023] Figure 5 is a schematic block diagram of a device for processing sensor information according to an embodiment of the present disclosure;

[0024] Figure 6Schematic block diagram of a processing device for sensor information according to another embodiment of the present disclosure;

[0025] Figure 7 Schematic block diagram of a processing device for sensor information according to yet another embodiment of the present disclosure;

[0026] Figure 8 Schematic block diagram of a processing device for sensor information according to yet another embodiment of the present disclosure;

[0027] Figure 9 Schematic block diagram of a processing device for sensor information according to yet another embodiment of the present disclosure;

[0028] Figure 10 Schematic block diagram of a processing device for sensor information according to yet another embodiment of the present disclosure;

[0029] Figure 11 Block diagram of an electronic device for implementing the method for processing sensor information according to an embodiment of the present disclosure. Detailed implementation manners

[0030] The following describes exemplary embodiments of the present disclosure with reference to the accompanying drawings. Various details of the embodiments of the present disclosure are included to facilitate understanding, and they should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted below.

[0031] Figure 1 A method for processing sensor information according to an embodiment of the present disclosure. This method can be applied to a processing device for sensor information, and the device can be deployed in electronic devices such as terminal devices and servers. As Figure 1 shown, the method may include:

[0032] Step S110: Obtain configuration information of the sensor; wherein, the configuration information of the sensor is used to determine calibration parameters of the sensor;

[0033] Step S120: Display an imaging simulation diagram of the sensor in a target scenario, and the imaging simulation diagram includes an image of a calibration reference object. Wherein, the imaging simulation diagram is generated based on the calibration parameters and virtual scene data, and the virtual scene data includes three-dimensional data of the calibration reference object.

[0034] Exemplarily, the sensor may include a camera, a radar, etc., which can display perception results through imaging.

[0035] The imaging process of the sensor can be simulated using a mathematical model, which includes an external parameter model and an internal parameter model. Among them, the external parameter model can be used to represent the conversion process between the three-dimensional scene in the world coordinate system (i.e., the real world) and the three-dimensional scene in the camera coordinate system. The internal parameter model can be used to represent the conversion process between the image sensed by the sensor and the three-dimensional scene in the camera coordinate system. Since the parameters in the above mathematical model are often obtained through calibration, the parameters in the above mathematical model can be called the calibration parameters of the sensor. That is to say, in the embodiments of the present disclosure, the calibration parameters can include the parameters in the internal parameter model (i.e., the internal parameters) and the parameters in the external parameter model (i.e., the external parameters).

[0036] Exemplarily, the configuration information of the sensor can include one or more pieces of information for determining the internal parameters and the external parameters. For example, the configuration information can include the internal parameters and the external parameters themselves, or can include information such as the model of the sensor, the focal length, the installation position, etc. that can be used to obtain the internal parameters and the external parameters.

[0037] Exemplarily, the configuration information of the sensor can be obtained in a manner of interacting with the user. For example, the configuration information of the sensor is determined based on the user's input or selection. For example, an input box can be provided on the user interface for the user to input the internal parameters of the sensor and the three-dimensional coordinates of the installation position of the sensor. For another example, a selection box can be provided on the user interface for the user to select the installation position of the sensor from multiple installation positions with known three-dimensional coordinates. In the scenario of checking the layout of vehicle sensors, each installation position can be represented by a vehicle component in the target scene. For example, each installation position can be the side front of the vehicle B-pillar (the pillar between the driver's side window glass and the rear side window glass), the rear side of the vehicle body, etc.

[0038] Exemplarily, the target scene can refer to a physical scene with a pre-constructed three-dimensional scene model, such as scenes related to autonomous driving like a basement or a road. In the embodiments of the present disclosure, the virtual scene data can be the data corresponding to the three-dimensional scene model. By collecting data in a real physical scene in advance and modeling based on the collected data, a three-dimensional scene model of the target scene can be obtained, and thus the virtual scene data for carrying the three-dimensional scene model can be obtained. Exemplarily, the virtual scene data can include the three-dimensional coordinates of each position point in the target scene in the world coordinate system and the corresponding imaging values.

[0039] After obtaining the configuration information of the sensor and determining the corresponding calibration parameters, the electronic device can perform imaging simulation based on the calibration parameters and the virtual scene data, that is, determine the correspondence between each position point in the virtual scene data and each pixel in the image based on the calibration parameters, and project the imaging values in the virtual scene data onto the corresponding pixels in the image according to the correspondence, so as to obtain the imaging simulation diagram.

[0040] According to an embodiment of the present disclosure, the virtual scene data may include three-dimensional data of a calibration reference object. Correspondingly, the imaging simulation diagram obtained based on the calibration parameters and the virtual scene data includes an image of the calibration parameters. Thus, when the imaging simulation diagram is displayed, the calibration reference object can be presented to the user together.

[0041] Exemplarily, the calibration reference object can be used to characterize the calibration parameters. The calibration parameters can be parameters for calibrating the perception effect. For example, the calibration parameters can be the illumination height, the size of the blind area, and the size of the overlapping area with other sensors, etc. Specifically, when the calibration parameter is the illumination height, the calibration reference object can be a scale for reflecting the illumination height. When the calibration parameter is the overlapping area between the current sensor and other sensors, the calibration reference object can be a geometric body that covers the perception area of other sensors and has a specific texture. According to actual application requirements, the calibration reference object can also be other reference objects such as a checkerboard.

[0042] By using the method provided in the above embodiment, based on the obtained configuration information of the sensor, an imaging simulation diagram is generated to present the perception area of the sensor in the target scene in the form of imaging simulation. And, by setting the three-dimensional data of the calibration reference object in the virtual scene data of the target scene, an image of the calibration reference object is included in the imaging simulation diagram. Based on this, the user can determine whether the perception effect of the sensor under the current configuration meets the calibration requirements based on the imaging simulation diagram and the calibration reference object therein, which has the advantages of comprehensive, intuitive, and efficient calibration.

[0043] In some embodiments, devices for installing sensors such as vehicles and road infrastructure devices can be set in the three-dimensional scene model corresponding to the virtual scene data. When the configuration information of the sensor indicates that the sensor is installed on these devices, the imaging simulation diagram can truly reflect the perception effect when the perception area of the sensor is blocked by the components of these devices. For example, when the sensor is set on a vehicle, the occlusion situations of the sensor bracket, the engine hood, the body components, etc. on the vehicle will be reflected in the imaging simulation diagram.

[0044] Figure 2 Fig. shows a schematic diagram of an imaging simulation diagram according to an embodiment of the present disclosure. This imaging simulation diagram is an imaging simulation diagram of a rear side camera of a vehicle body. In this imaging simulation diagram, it can be seen that the rearview mirror 201 of the vehicle and the front side 202 of the vehicle block some areas in the target scene.

[0045] Figure 3 Fig. shows a schematic diagram of another imaging simulation diagram according to an embodiment of the present disclosure. This imaging simulation diagram is an imaging simulation diagram of a front side camera on the B-pillar of the vehicle. In this imaging simulation diagram, it can be seen that there is an opening in front of the front side camera on the B-pillar of the vehicle, and the edge 301 of the opening blocks some areas in the target scene.

[0046] It can be seen that by means of imaging simulation for sensor calibration, all occlusion factors can be fully considered, thereby avoiding design errors and repetitive work.

[0047] Optionally, in some embodiments of the present disclosure, the method for processing sensor information may further include:

[0048] Obtaining configuration information of calibration parameters; wherein, the configuration information of calibration parameters is used to determine three-dimensional data of a calibration reference object corresponding to the calibration parameters, so as to obtain virtual scene data based on the three-dimensional data of the calibration reference object and the three-dimensional data of the target scene.

[0049] Exemplarily, the configuration information of the calibration parameters can be obtained by interacting with the user. For example, the configuration information of the calibration parameters is determined based on the user's input. The form of this configuration information is related to the type of calibration parameters. For example, if the calibration parameter is the illumination height at a certain distance in front of the vehicle, the corresponding configuration information may be the distance value in front of the vehicle. Another example is that if the calibration parameter is the size of the overlapping area between the currently calibrated sensor and other sensors, the corresponding configuration information may be information for determining the sensing area of other sensors, such as the installation position and internal parameters of other sensors.

[0050] Exemplarily, according to the configuration information of the calibration parameters, the position of the calibration reference object can be determined. This position is represented by three-dimensional data, including the three-dimensional coordinates of each point in the calibration reference object; then, based on the three-dimensional data of the calibration reference object and the three-dimensional data of the target scene, virtual scene data is obtained. Among them, the three-dimensional data of the target scene includes the three-dimensional coordinates of each object in the three-dimensional scene model of the target scene. Exemplarily, the calibration reference object can be set in the three-dimensional scene model according to this three-dimensional data, so as to obtain the above virtual scene data based on the three-dimensional data of the calibration reference object and the three-dimensional data of the target scene. For example, if the configuration information of the illumination height indicates that the illumination height is the illumination height 3 meters in front of the vehicle, then based on this configuration information, it can be determined that the scale is located 3 meters in front of the vehicle, so as to determine the three-dimensional data of the scale, and then add the scale to the three-dimensional scene model according to this three-dimensional data, thereby obtaining the corresponding virtual scene data.

[0051] Figure 4 FIG. shows a schematic diagram of a three-dimensional scene model corresponding to virtual scene data according to an embodiment of the present disclosure. As Figure 4 shown, in the three-dimensional scene model, a vehicle 401 is provided. If the configuration information of the calibration parameters is used to configure scales at 3 meters in front of and behind the vehicle 401 and at 3 meters on both sides of the vehicle 401, then the three-dimensional scene model includes as Figure 4The scale bars 402, 403, 404, and 405 shown. Through these scale bars, the illumination height of the sensor at 3 meters in front of and behind the vehicle and at 3 meters on both sides of the vehicle can be visually seen.

[0052] In the above embodiments, the calibration parameters are configurable, and the user can input configuration information according to actual needs to customize the calibration reference for characterizing the calibration parameters, so that the imaging simulation diagram shows information consistent with the actual needs, avoiding the repetitive labor of the user and thus improving the calibration efficiency.

[0053] Optionally, in some embodiments of the present disclosure, the virtual scene data is also configurable. For example, the method for processing sensor information may further include: obtaining configuration information of the virtual scene data, and this configuration information can be used to select the virtual scene data of the target scene from the virtual scene data of multiple pre-stored scenes. For example, this configuration information may be the identifier of the target scene, such as Garage 1, Garage 2, Road Section 1, etc.

[0054] Optionally, in some embodiments of the present disclosure, various types of parameters can be controlled by configuration files. Specifically, on the basis of the foregoing embodiments, the method for processing sensor information may further include: determining a sensor configuration file based on the configuration information of the sensor, where the sensor configuration file is used to characterize the calibration parameters of the sensor; generating an imaging simulation diagram based on the scene configuration file and the sensor configuration file, where the scene configuration file is used to characterize the virtual scene data.

[0055] That is to say, the calibration parameters of the sensor and the virtual scene data are both carried by the corresponding configuration files. In the case of changing the configuration information of the sensor or switching scenes, the configuration of the imaging simulation can be quickly adjusted by modifying or switching the configuration files, so as to quickly generate an imaging simulation diagram corresponding to the new configuration and improve the calibration efficiency.

[0056] Optionally, in some embodiments of the present disclosure, multiple display modes can be used to display the sensing effect of the sensor. Specifically, in the method for processing sensor information, displaying the imaging simulation diagram of the sensor may include:

[0057] In response to a selection operation for the i-th display mode among N display modes, displaying the imaging simulation diagram corresponding to the i-th display mode of the sensor in the target scene; where the imaging simulation diagram corresponding to the i-th display mode is obtained based on the calibration parameters and the virtual scene data corresponding to the i-th display mode, N is an integer greater than or equal to 2, and i is a positive integer less than or equal to N.

[0058] Among them, the i-th display mode can be any one of the N display modes. That is to say, in response to a user's selection operation of a certain display mode, the electronic device can perform imaging simulation using the virtual scene data corresponding to the display mode, so as to obtain an imaging simulation diagram corresponding to the display mode and display the imaging simulation diagram.

[0059] In the embodiments of the present disclosure, for the same target scene, virtual scene data corresponding to different display modes is set. For example, multiple virtual physical data corresponding to different display modes are set in the same garage or the same road section. Optionally, the presentation effects of the virtual scene data corresponding to different display modes on the target scene are different.

[0060] The above embodiments support displaying the imaging simulation diagram in multiple display modes, so that the user can select the corresponding display mode according to actual needs, which is convenient for the user to quickly check the sensing effect of the sensor and improve the checking efficiency.

[0061] In an example, the method for processing sensor information may further include: obtaining a first scene map of the target scene based on a panoramic image collected by a panoramic camera in the target scene; obtaining virtual scene data corresponding to the j-th display mode among the N display modes based on the first scene map; where j is a positive integer less than or equal to N.

[0062] Exemplarily, the panoramic image collected by the panoramic camera can be used as the first scene map of the target scene. The virtual scene data obtained based on the first scene map can contain more real and complete scene information, so that the imaging simulation diagram can reflect a more real and complete sensing area of the sensor and improve the accuracy of sensor checking.

[0063] In an example, the method for processing sensor information may further include: stitching at least two images collected from different orientations by at least two cameras in the target scene to obtain a second scene map of the target scene; obtaining virtual scene data corresponding to the k-th display mode among the N display modes based on the second scene map; where k is a positive integer less than or equal to N.

[0064] Exemplarily, fisheye stitching can be performed on at least two images, and the fisheye stitched image can be used as the second scene map of the target scene.

[0065] Optionally, the at least two cameras can be deployed on a specific vehicle. For example, during the vehicle development process, a second vehicle similar to the first vehicle to be developed can be selected, and the at least two cameras can be deployed on the second vehicle, so as to obtain a second scene map for establishing a three-dimensional scene model by using the images collected by the at least two cameras. Virtual scene data is obtained by using the second scene map, and then when arranging and designing the sensors on the first vehicle, the virtual scene data is used for imaging simulation. Since the virtual scene data is collected by using the cameras on a similar vehicle model, the imaging simulation map obtained based on this method is closer to the sensing result of the sensors on the first vehicle.

[0066] It can be seen that in the above example, the deployment positions of the at least two cameras can be flexibly set according to the deployment positions of the sensors, so that the virtual scene data obtained based on the second scene map better meets the application requirements. Correspondingly, the imaging simulation map is also closer to the sensing result of the sensors in actual application, thereby improving the accuracy of sensor calibration.

[0067] Optionally, in some embodiments of the present disclosure, the method for processing sensor information may further include: obtaining vehicle speed information; in response to the vehicle speed information, displaying a driving simulation video of the sensor; wherein the driving simulation video includes M imaging simulation maps respectively corresponding to M driving moments, and M is an integer greater than or equal to 2.

[0068] Exemplarily, the vehicle speed information may be the speed of the vehicle where the sensor is located. Therefore, the vehicle speed information is the moving speed of the sensor.

[0069] Optionally, the vehicle speed information can be obtained by interacting with the user. For example, the vehicle speed information is input by the user.

[0070] According to the above embodiments, when the user inputs vehicle speed information, multiple driving moments can be determined according to the vehicle speed information, and for each driving moment, a corresponding imaging simulation map is generated respectively, so as to obtain a corresponding driving simulation video. That is to say, driving simulation based on vehicle speed is supported, which is more conducive to the user to calibrate the sensing effect of the sensor and improve the calibration integrity.

[0071] Optionally, on the basis of the above embodiments, the method for processing sensor information may further include: determining the external parameters of the sensor at each of the M driving moments based on the vehicle speed information and the configuration information of the sensor; obtaining the imaging simulation map corresponding to each driving moment based on the internal parameters of the sensor, the external parameters at each driving moment, and the virtual scene data.

[0072] Under the influence of vehicle speed, the position of the sensor is different at different driving times. Therefore, the extrinsic parameters of the sensor also change accordingly. According to the above embodiments, by determining the extrinsic parameters of the sensor at each driving time based on the vehicle speed information, corresponding imaging simulation diagrams can be generated using the extrinsic parameters at each driving time, improving the authenticity of the driving simulation video and facilitating more accurate verification of the sensor layout.

[0073] It can be seen that according to the method of the embodiments of the present disclosure, based on the acquired configuration information of the sensor, an imaging simulation diagram is generated to present the sensing area of the sensor in the form of imaging simulation. Moreover, by setting the three-dimensional data of the verification reference object in the virtual scene data, the imaging simulation diagram includes the image of the verification reference object. Based on this, the user can determine whether the sensing effect of the sensor under the current configuration meets the verification requirements based on the imaging simulation diagram and the verification reference object therein, which has the advantages of comprehensive, intuitive, and efficient verification.

[0074] In the technical solution of the present disclosure, the acquisition, storage, and application of the user's personal information involved all comply with the provisions of relevant laws and regulations and do not violate public order and good customs.

[0075] According to an embodiment of the present disclosure, the present disclosure also provides a processing device for sensor information for implementing the above method. Figure 5 The schematic block diagram of a processing device for sensor information provided by an embodiment of the present disclosure is shown.

[0076] A configuration information acquisition module 510, configured to acquire the configuration information of the sensor; wherein, the configuration information of the sensor is used to determine the calibration parameters of the sensor;

[0077] An image display module 520, configured to display an imaging simulation diagram of the sensor in a target scene, where the imaging simulation diagram includes the image of the verification reference object; wherein, the imaging simulation diagram is generated based on the calibration parameters and the virtual scene data of the target scene, and the virtual scene data includes the three-dimensional data of the verification reference object.

[0078] In some embodiments, the configuration information acquisition module 510 is further configured to:

[0079] Acquire the configuration information of the verification parameters; wherein, the configuration information of the verification parameters is used to determine the three-dimensional data of the verification reference object corresponding to the verification parameters, so as to obtain the virtual scene data based on the three-dimensional data of the verification reference object and the three-dimensional data of the target scene.

[0080] In some embodiments, on the Figure 5 basis, as Figure 6 shown, the processing device for sensor information further includes:

[0081] A configuration information processing module 610 is configured to determine a sensor configuration file based on the configuration information of a sensor; wherein, the sensor configuration file is used to characterize the calibration parameters of the sensor.

[0082] A first image generation module 620 is configured to generate an imaging simulation diagram based on a scene configuration file and the sensor configuration file; wherein, the scene configuration file is used to characterize virtual scene data.

[0083] In some embodiments, the image display module 520 is specifically configured to:

[0084] In response to a selection operation for the i-th display mode among N display modes, display the imaging simulation diagram corresponding to the i-th display mode of the sensor in a target scene; wherein, the imaging simulation diagram corresponding to the i-th display mode is obtained based on the calibration parameters and the virtual scene data corresponding to the i-th display mode, N is an integer greater than or equal to 2, and i is a positive integer less than or equal to N.

[0085] In some embodiments, on the basis of Figure 5 , as Figure 7 shown, the processing device for sensor information further includes:

[0086] A first image processing module 710 is configured to obtain a first scene map of the target scene based on a panoramic image collected by a panoramic camera in the target scene.

[0087] A first map processing module 720 is configured to obtain virtual scene data corresponding to the j-th display mode among N display modes based on the first scene map; wherein, j is a positive integer less than or equal to N.

[0088] In some embodiments, on the basis of Figure 7 , as Figure 8 shown, the processing device for sensor information further includes:

[0089] A second image processing module 810 is configured to splice at least two images collected from different orientations by at least two cameras in the target scene to obtain a second scene map of the target scene.

[0090] A second map processing module 820 is configured to obtain virtual scene data corresponding to the k-th display mode among N display modes based on the second scene map; wherein, k is a positive integer less than or equal to N.

[0091] In some embodiments, on the basis of Figure 5 , as Figure 9 shown, it further includes:

[0092] A vehicle speed information acquisition module 910 is configured to acquire vehicle speed information.

[0093] A video display module 920, configured to display a driving simulation video of a sensor in response to vehicle speed information; wherein, the driving simulation video includes M imaging simulation diagrams respectively corresponding to M driving moments, and M is an integer greater than or equal to 2.

[0094] In some embodiments, based on Figure 9 , as Figure 10 shown, it further includes:

[0095] A vehicle speed information processing module 1010, configured to determine the extrinsic parameters of the sensor at each of the M driving moments based on the vehicle speed information and the configuration information of the sensor;

[0096] A second image generation module 1020, configured to obtain the imaging simulation diagram corresponding to each driving moment based on the intrinsic parameters of the sensor, the extrinsic parameters at each driving moment, and the virtual scene data.

[0097] For the specific functions and examples of the modules and sub - modules of the device according to the embodiments of the present disclosure, reference may be made to the relevant descriptions of the corresponding steps in the above - mentioned method embodiments, which will not be elaborated herein.

[0098] According to the embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0099] Figure 11 FIG. shows a schematic block diagram of an exemplary electronic device 1100 that can be used to implement the embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, a personal digital assistant, a cellular phone, a smart phone, a wearable device, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0100] As Figure 11 shown, the device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read - only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the device 1100 can also be stored. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other through a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0101] Multiple components in device 1100 are connected to I / O interface 1105, including: input unit 1106, such as a keyboard, mouse, etc.; output unit 1107, such as various types of displays, speakers, etc.; storage unit 1108, such as a disk, optical disc, etc.; and communication unit 1109, such as a network card, modem, wireless communication transceiver, etc. Communication unit 1109 allows device 1100 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0102] Computing unit 1101 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of computing unit 1101 include but are not limited to a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Computing unit 1101 executes the various methods and processes described above, such as the method for processing sensor information. For example, in some embodiments, the method for processing sensor information can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed onto device 1100 via ROM 1102 and / or communication unit 1109. When the computer program is loaded into RAM 1103 and executed by computing unit 1101, one or more steps of the method for processing sensor information described above can be executed. Alternatively, in other embodiments, computing unit 1101 can be configured to execute the method for processing sensor information in any other suitable manner (e.g., by means of firmware).

[0103] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special-purpose or general-purpose programmable processor, and can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0104] The program code for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general purpose computer, a special purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program codes cause the functions / operations specified in the flowchart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0105] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0106] In order to provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0107] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), and the Internet.

[0108] A computer system can include a client and a server. The client and the server are generally far from each other and typically interact through a communication network. The client - server relationship is created by computer programs running on the respective computers and having a client - server relationship with each other. The server can be a cloud server, a server of a distributed system, or a server incorporating blockchain.

[0109] It should be understood that various forms of the processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in this disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this is not limited herein.

[0110] The above - described specific embodiments do not constitute a limitation on the protection scope of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub - combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure shall be included within the protection scope of this disclosure.

Claims

1. A method for processing sensor information, including: obtaining configuration information of a sensor; wherein, the configuration information of the sensor is used to determine calibration parameters of the sensor; obtaining configuration information of a calibration parameter; wherein, the configuration information of the calibration parameter is used to determine three-dimensional data of a calibration reference object corresponding to the calibration parameter, so as to obtain virtual scene data of the target scene based on the three-dimensional data of the calibration reference object and the three-dimensional data of the target scene; the calibration reference object is used to represent the calibration parameter, the calibration parameter is a parameter for calibrating a sensing effect, and the calibration parameter includes illumination height, blind area size, or size of an overlapping area with other sensors; displaying an imaging simulation diagram of the sensor in the target scene, where the imaging simulation diagram includes an image of the calibration reference object; wherein, the imaging simulation diagram is generated based on the calibration parameters and the virtual scene data of the target scene, and the virtual scene data includes the three-dimensional data of the calibration reference object.

2. The method according to claim 1, further including: determining a sensor configuration file based on the configuration information of the sensor; wherein, the sensor configuration file is used to represent the calibration parameters of the sensor; generating the imaging simulation diagram based on a scene configuration file and the sensor configuration file; wherein, the scene configuration file is used to represent the virtual scene data.

3. The method according to claim 1 or 2, wherein, the displaying the imaging simulation diagram of the sensor in the target scene includes: responding to a selection operation for the i-th display mode among N display modes, and displaying the imaging simulation diagram corresponding to the i-th display mode of the sensor in the target scene; wherein, the imaging simulation diagram corresponding to the i-th display mode is obtained based on the calibration parameters and the virtual scene data corresponding to the i-th display mode, N is an integer greater than or equal to 2, and i is a positive integer less than or equal to N.

4. The method according to claim 3, further including: obtaining a first scene texture map of the target scene based on a panoramic image collected by a panoramic camera in the target scene; obtaining virtual scene data corresponding to the j-th display mode among the N display modes based on the first scene texture map; wherein, j is a positive integer less than or equal to N.

5. The method according to claim 3, further including: stitching at least two images collected from different orientations by at least two cameras in the target scene to obtain a second scene texture map of the target scene; obtaining virtual scene data corresponding to the k-th display mode among the N display modes based on the second scene texture map; wherein, k is a positive integer less than or equal to N.

6. The method according to claim 1 or 2, further including: obtaining vehicle speed information; responding to the vehicle speed information, and displaying a driving simulation video of the sensor; wherein, the driving simulation video includes M imaging simulation diagrams respectively corresponding to M driving moments, and M is an integer greater than or equal to 2.

7. The method according to claim 6, further including: Based on the vehicle speed information and the configuration information of the sensor, determine the extrinsic parameters of the sensor at each of the M driving moments. Based on the intrinsic parameters of the sensor, the extrinsic parameters at each driving moment, and the virtual scene data, obtain the imaging simulation diagram corresponding to each driving moment.

8. A processing device for sensor information comprising: A configuration information acquisition module, configured to acquire the configuration information of the sensor and the configuration information of the calibration parameters; wherein, the configuration information of the sensor is used to determine the calibration parameters of the sensor; the configuration information of the calibration parameters is used to determine the three-dimensional data of the calibration reference object corresponding to the calibration parameters, so as to obtain the virtual scene data of the target scene based on the three-dimensional data of the calibration reference object and the three-dimensional data of the target scene; the calibration reference object is used to represent the calibration parameters, the calibration parameters are parameters for calibrating the perception effect, and the calibration parameters include the illumination height, the blind area size, or the size of the overlapping area with other sensors. An image display module, configured to display the imaging simulation diagram of the sensor in the target scene, where the imaging simulation diagram includes an image of the calibration reference object; wherein, the imaging simulation diagram is generated based on the calibration parameters and the virtual scene data of the target scene, and the virtual scene data includes the three-dimensional data of the calibration reference object.

9. The device according to claim 8, further comprising: A configuration information processing module, configured to determine a sensor configuration file based on the configuration information of the sensor; wherein, the sensor configuration file is used to represent the calibration parameters of the sensor. A first image generation module, configured to generate the imaging simulation diagram based on the scene configuration file and the sensor configuration file; wherein, the scene configuration file is used to represent the virtual scene data.

10. The device according to claim 8 or 9, wherein, the image display module is specifically configured to: In response to a selection operation for the i-th display mode among the N display modes, display the imaging simulation diagram corresponding to the i-th display mode of the sensor in the target scene; wherein, the imaging simulation diagram corresponding to the i-th display mode is obtained based on the calibration parameters and the virtual scene data corresponding to the i-th display mode, N is an integer greater than or equal to 2, and i is a positive integer less than or equal to N.

11. The device according to claim 10, further comprising: A first image processing module, configured to obtain a first scene texture map of the target scene based on the panoramic image collected by the panoramic camera in the target scene. A first texture processing module, configured to obtain the virtual scene data corresponding to the j-th display mode among the N display modes based on the first scene texture map; wherein, j is a positive integer less than or equal to N.

12. The device according to claim 10, further comprising: A second image processing module, configured to splice at least two images collected from different orientations by at least two cameras in the target scene to obtain a second scene texture map of the target scene. A second texture mapping processing module, configured to obtain virtual scene data corresponding to the k-th display mode among the N display modes based on the second scene texture map, where k is a positive integer less than or equal to N.

13. The apparatus according to claim 8 or 9, further comprises: A vehicle speed information acquisition module, configured to acquire vehicle speed information; A video display module, configured to display a driving simulation video of the sensor in response to the vehicle speed information, where the driving simulation video includes M imaging simulation diagrams respectively corresponding to M driving times, and M is an integer greater than or equal to 2.

14. The apparatus according to claim 13, further comprises: A vehicle speed information processing module, configured to determine the external parameters of the sensor at each of the M driving times based on the vehicle speed information and the configuration information of the sensor; A second image generation module, configured to obtain the imaging simulation diagram corresponding to each of the driving times based on the internal parameters of the sensor, the external parameters at each driving time, and the virtual scene data.

15. An electronic device, comprises: At least one processor; and A memory communicatively connected to the at least one processor, where The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method according to any one of claims 1-7.

16. A non-transitory computer-readable storage medium storing computer instructions, where The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

17. A computer program product, comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-7.

Images