Virtual object simulation method, device, equipment and storage medium

Abstract

The present disclosure provides a virtual object simulation method, device, equipment and storage medium, which relates to the field of artificial intelligence technology, especially to the field of autonomous driving. The specific implementation scheme is: determining a virtual environment scene; determining the collection information for the target object, the collection information is obtained by using a sensor to collect the target object in a predetermined real scene; determining the perception information for the target object according to the virtual environment scene, the collection information and the position information of the reference object; and generating a virtual object corresponding to the target object in the virtual environment scene according to the perception information; wherein the position information of the reference object includes: the first position of the reference object in the virtual environment scene coordinate system, and the second position of the reference object in the sensor coordinate system.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to the field of autonomous driving. More specifically, the present disclosure provides a virtual object simulation method, a virtual object simulation device, an electronic device, a storage medium, a computer program product, and a virtual object simulation device. Background Art

[0002] Perception is an important part of autonomous driving. Through perception, autonomous vehicles can determine whether there are obstacles in the surrounding environment, the type of obstacles, the distance of obstacles, and other information, and make different decisions based on this information. Summary of the invention

[0003] The present disclosure provides a virtual object simulation method, a virtual object simulation apparatus, an electronic device, a storage medium, a computer program product, and a virtual object simulation device.

[0004] According to one aspect of the present disclosure, a virtual object simulation method is provided, comprising: determining a virtual environment scene; determining acquisition information for a target object, the acquisition information being acquired by using a sensor to acquire information about the target object in a predetermined real scene; determining perception information for the target object based on the virtual environment scene, the acquisition information, and position information of a reference object; and generating a virtual object corresponding to the target object in the virtual environment scene based on the perception information; wherein the position information of the reference object comprises: a first position of the reference object in a virtual environment scene coordinate system, and a second position of the reference object in a sensor coordinate system.

[0005] According to another aspect of the present disclosure, a virtual object simulation device is provided, comprising a first determination module, a second determination module, a third determination module and a first generation module. The first determination module is used to determine a virtual environment scene; the second determination module is used to determine acquisition information for a target object, the acquisition information being acquired by using a sensor to acquire the target object in a predetermined real scene; the third determination module is used to determine perception information for the target object based on the virtual environment scene, the acquisition information and the position information of the reference object; the first generation module is used to generate a virtual object corresponding to the target object in the virtual environment scene based on the perception information; wherein the position information of the reference object includes: a first position of the reference object in the virtual environment scene coordinate system, and a second position of the reference object in the sensor coordinate system.

[0006] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided by the present disclosure.

[0007] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the method provided by the present disclosure.

[0008] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method provided by the present disclosure is implemented.

[0009] According to another aspect of the present disclosure, a virtual object simulation device is provided, including the electronic device provided by the present disclosure and a sensing mechanism, the sensing mechanism including a rack and at least one sensor installed on the rack, each sensor being used to detect a target object located around the rack to obtain collected information.

[0010] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0012] Figure 1 is a schematic diagram of an application scenario of a virtual object simulation method and device according to an embodiment of the present disclosure;

[0013] Figure 2 is a schematic flow chart of a virtual object simulation method according to an embodiment of the present disclosure;

[0014] Figure 3 is a schematic diagram of a virtual object simulation method according to an embodiment of the present disclosure;

[0015] Figure 4 is a schematic structural block diagram of a virtual object simulation device according to an embodiment of the present disclosure;

[0016] Figure 5 is a schematic structural block diagram of a virtual object simulation device according to an embodiment of the present disclosure; and

[0017] Figure 6 It is a structural block diagram of an electronic device used to implement the virtual object simulation method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0018] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0019] To develop perception algorithms, it is often necessary to carry out a large amount of data collection, scenario testing, simulation and other work for various scenarios.

[0020] In some technical solutions, a virtual simulation scene can be built through a game engine, and both sensor data and scene data are obtained through the method built in the game engine. It is understandable that with this solution, there is some gap between the data obtained from the game engine and the real data, which leads to poor verification effect.

[0021] The embodiments of the present disclosure provide a simulation method for perceiving hardware in the loop. Since the collected information is obtained by using sensors to collect real objects in a predetermined real scene, compared with the simulation method of a game engine, the collected information of the embodiments of the present disclosure is more realistic, thereby obtaining a more realistic simulation effect.

[0022] The technical solution provided by the present disclosure will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] Figure 1 It is a schematic diagram of an application scenario of a virtual object simulation method and device according to an embodiment of the present disclosure.

[0024] It should be noted that Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0025] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a sensor 101 , an electronic device 102 , and a display device 103 .

[0026] The sensor 101 may include at least one of a laser radar, a millimeter wave radar, an ultrasonic radar, an image acquisition device, and an inertial navigation device. During the simulation process, the sensor 101 may be installed on a rack, and the rack may be placed in a predetermined real scene, such as indoors.

[0027] The electronic device 102 is electrically connected to the sensor 101 to obtain the collected information of the sensor 101 , such as image data, point cloud data, etc. collected by the sensor 101 .

[0028] The display device 103 can display a human-computer interaction interface to facilitate user operation. A virtual environment scene can be displayed in the human-computer interaction interface, and the virtual environment scene can correspond to map data. In addition, a virtual reference object corresponding to a reference object can be generated in the virtual environment scene, and the reference object can be, for example, a rack or a sensor 101. A virtual object corresponding to a target object can also be generated in the virtual environment scene, and the target object can represent obstacles such as guardrails, pedestrians, and vehicles.

[0029] During the simulation process, the electronic device 102 can determine the first position of the reference object in the virtual environment scene according to the virtual environment information, map data, and virtual reference objects. The electronic device 102 can receive the image data, point cloud data, and other collected information collected by the sensor 101, and use the perception algorithm or other methods to process the collected information to obtain the position of the target object in the coordinate system of the sensor 101, and can also obtain the second position of the reference object in the coordinate system of the sensor 101. The electronic device 102 can use the perception algorithm to process the collected information, the position information of the reference object, and the virtual environment scene to obtain the perception information, and generate the virtual object based on the perception information.

[0030] It should be noted that the virtual object simulation method provided in the embodiment of the present disclosure can generally be executed by the electronic device 102. Accordingly, the virtual object simulation device provided in the embodiment of the present disclosure can generally be set in the electronic device 102. The virtual object simulation method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the electronic device 102 and can communicate with the sensor 101, the display device 103 and / or the electronic device 102. Correspondingly, the virtual object simulation device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the electronic device 102 and can communicate with the sensor 101, the display device 103 and / or the electronic device 102.

[0031] It should be understood that Figure 1 The number of sensors, electronic devices and display devices in the embodiment is only illustrative. Any number of sensors, electronic devices and display devices may be provided according to implementation requirements.

[0032] Figure 2 is a schematic flow chart of a virtual object simulation method according to an embodiment of the present disclosure.

[0033] like Figure 2 As shown, the virtual object simulation method 200 may include operations S210 to S240.

[0034] In operation S210 , a virtual environment scene is determined.

[0035] For example, virtual environment scenes such as unprotected left turns, urban roads, and intersections can be pre-built according to needs, and the virtual environment scenes can include road information. The scene selector can be used to select the virtual environment scene to be used from the pre-built virtual environment scenes. For example, after receiving the selection instruction, the scene selector selects the virtual environment scene to be used according to the scene identifier included in the selection instruction.

[0036] In operation S220, collection information for the target object is determined, where the collection information is obtained by collecting the target object in a predetermined real scene using a sensor.

[0037] For example, the target objects can be set around the sensing mechanism, and the target objects can be used to simulate obstacles such as pedestrians, vehicles, guardrails, cones, etc. For example, if the sensing mechanism is set up in a laboratory for simulation testing, physical models of various shapes can be placed in the laboratory and used as target objects.

[0038] For example, the sensing mechanism can be used to collect data from the target object to obtain collected information.

[0039] For example, the sensing mechanism may include a rack and at least one sensor. The rack may be placed in a predetermined real scene, such as a laboratory for simulation testing. The sensor is mounted on the rack, and the sensor may include at least one of a laser radar, a millimeter wave radar, an ultrasonic radar, an image acquisition device, and an inertial navigation device. The sensor is used to detect target objects located around the rack to obtain acquisition information, such as the information acquired by the image acquisition device includes image data, and the information acquired by the laser radar includes point cloud data. The sensor may also send the acquired information to an electronic device.

[0040] In operation S230, perception information for the target object is determined according to the virtual environment scene, the collected information, and the position information of the reference object. The position information of the reference object includes: a first position of the reference object in the virtual environment scene coordinate system, and a second position of the reference object in the sensor coordinate system.

[0041] For example, the virtual environment scene may correspond to map data, and the location information in the map data corresponding to any location in the virtual environment scene may be determined through the map data.

[0042] For example, the sensing mechanism can be used as a reference object, or a sensor or a rack in the sensing mechanism can be used as a reference object. The first position of the reference object in the virtual environment scene coordinate system can be determined in combination with the map data. The relative position relationship between the target object and the sensor and the position of the target object in the sensor coordinate system can be determined based on the information collected by the sensor, thereby obtaining the second position of the reference object in the sensor coordinate system.

[0043] For example, the perception information may include the position, speed, orientation, etc. of the target object, and a bounding box may be used to represent the position. It should be understood that since the sensor coordinate system and the virtual environment scene coordinate system can be transformed, each type of perception information may include first information in the sensor coordinate system and second information in the virtual environment scene coordinate system. For example, the position of the target object may include a third position of the target object in the virtual environment scene coordinate system, and may also include a fourth position of the target object in the sensor coordinate system. In addition, when determining the position of the target object, the perception algorithm may first process the image data, point cloud data, etc. collected by the sensor to obtain the fourth position of the target object in the sensor coordinate system, and then transform the fourth position according to the sensor coordinate system and the virtual environment scene coordinate system to obtain the third position of the target object in the virtual environment scene coordinate system.

[0044] In the actual simulation process, the perception algorithm can be used to process the virtual environment scene, the collected information, and the position information of the reference object to obtain the perception information. The perception algorithm is an algorithm for realizing autonomous driving. For example, the perception algorithm can process the collected information to obtain information such as the type of the target object and the position of the target object. The present disclosure embodiment does not limit the perception algorithm.

[0045] It should be understood that for an autonomous driving vehicle, a sensor can be used to detect the GPS (Global Positioning System) coordinates of the vehicle, and the GPS coordinates, image data collected by the sensor, point cloud data, etc. can be input into the perception algorithm, and the perception algorithm outputs perception information. In the disclosed embodiment, the first position of the reference object in the virtual environment scene coordinate system can replace the above-mentioned GPS coordinates and input into the perception algorithm, and the collected information can also be input into the perception algorithm, and the perception algorithm outputs perception information.

[0046] In operation S240, a virtual object corresponding to the target object is generated in the virtual environment scene according to the perception information.

[0047] For example, a virtual object may be generated at a third position in the virtual environment scene. For example, in a predetermined real scene, the target object is located 10 meters in front of the reference object. In the virtual environment scene, the virtual object may be generated 10 meters in front of the virtual reference object representing the reference object. The reference object may include a sensing mechanism. In addition, the state of the virtual object may be determined based on the direction, speed, etc. in the sensing information. For example, if the target object in the real scene faces due north, the virtual object may face due north in the virtual environment scene.

[0048] According to the technical solution provided by the embodiment of the present disclosure, sensor hardware-in-the-loop simulation is realized through virtual environment scenes, reference object position information and sensor collection information. Since the collection information is obtained by using sensors to collect real objects in a predetermined real scene, the collection information is more realistic, thereby obtaining a more realistic simulation effect.

[0049] In addition, after the virtual object is generated, the generation result can be displayed in the human-computer interaction interface, so that the user can intuitively determine whether the perception algorithm is accurate. For example, if the target object in the room moves towards the reference object, but in the virtual environment scene, the virtual object moves away from the virtual reference object corresponding to the reference object, or the number of target objects placed in the room is different from the number of generated virtual objects, it can be determined that the perception algorithm is inaccurate and needs to be debugged.

[0050] In some embodiments, sensors may be used to collect data, and then the perception algorithm may be verified online in a simulation scenario. The data collected by the sensors, indoor positioning, etc. may also be stored as data packets, and the perception algorithm may be verified offline through the saved data packets.

[0051] According to another embodiment of the present disclosure, the method may further include the following operations: in response to receiving a generation instruction, generating a virtual reference object in the virtual environment scene according to the generation instruction, and then determining a position of the virtual reference object in the virtual environment scene according to map data corresponding to the virtual environment scene as the first position.

[0052] For example, a virtual environment scene can be displayed using a human-computer interaction interface. For example, a user can select a predetermined position in the human-computer interaction interface according to needs. In response to receiving the selected predetermined position, the virtual environment scene generates a virtual reference object for representing the reference object in the virtual environment scene. In addition, the predetermined position may include a starting position and an end position, and the virtual reference object can move from the starting position to the end position along a predetermined path in the virtual environment scene.

[0053] For example, the virtual environment scene has a corresponding relationship with the map data. For example, the starting point of a lane line in the virtual environment scene corresponds to a position information in the map data. Therefore, through the position of the virtual reference object in the virtual environment scene and the corresponding relationship between the virtual environment scene and the map data, the position information of the virtual reference object can be obtained from the map data, and the position information is the second position mentioned above. After determining the second position, the second position can also be sent to the electronic device through the subscription and publishing mechanism of the message.

[0054] According to the technical solution provided by the embodiment of the present disclosure, since the virtual reference object can be placed according to actual needs, and the second position of the virtual reference object will affect the third position of the virtual object, the third position of the virtual object in the virtual environment scene can be adjusted by the virtual reference object, so that the user can intuitively know the influence of the position of the virtual reference object on the virtual object.

[0055] According to another embodiment of the present disclosure, the above-mentioned operation of determining the collection information for the target object may include the following operations: in response to receiving a start instruction, turning on a fault switch related to the sensor, obtaining original collection information when the fault switch is turned on, and then determining the collection information based on the original collection information.

[0056] For example, the user operates on the human-computer interaction interface, and the operation can trigger an opening instruction. The electronic device receives the opening instruction and turns on the fault switch. The fault switch can be a program, and executing the program can simulate the effect of a sensor failure.

[0057] For example, after the fault switch is turned on, the internal and external reference files of the sensor can be adjusted to make the sensor's posture incorrect. For another example, the data collected by the laser radar needs to be transmitted to the electronic device using Ethernet. After the fault switch is turned on, the Ethernet can be disconnected to achieve the effect of disconnecting the laser radar. For another example, the electronic device needs to use rule files and soft link files to identify and read the image data collected by the camera. After the fault switch is turned on, the rule files and / or soft link files can be deleted to achieve the effect of causing the camera to malfunction.

[0058] The disclosed embodiments use software to simulate the effect of a sensor failure, thereby testing the impact of the sensor failure on the perception algorithm.

[0059] In some embodiments, all original collected information may be determined as the collected information.

[0060] In other embodiments, part of all the original collected information may be deleted, and the remaining original collected information may be determined as the collected information.

[0061] In one example, the number of original acquisition information is at least one, and each original acquisition information includes a first timestamp for indicating the acquisition time. The above method may also include the following operations: determining a second timestamp corresponding to each original acquisition information, the second timestamp indicating the time when the electronic device receives the original acquisition information. Next, for each original acquisition information, determine whether the first timestamp and the second timestamp meet the first predetermined condition, if so, delete the original acquisition information; if not, determine the original acquisition information as the acquisition information. The first predetermined condition may be that the interval between the first timestamp and the second timestamp is greater than or equal to the first predetermined time length, and the first predetermined time length may be, for example, 10 milliseconds.

[0062] For example, if the collected data detected by the sensor needs to be sent to the electronic device for data processing, the time base of the sensor and the time base of the electronic device can be made the same. Then compare the first timestamp of the data collected by the sensor and the second timestamp of the data received by the electronic device to obtain the first time interval between the two. If the first interval duration is greater than the first predetermined duration, it means that the timeliness of the data is low, so the data can be not used for calculation. If the first interval duration is less than or equal to the first predetermined duration, the original collected information can be determined as the collected information, and the collected information can be used to determine the perception information.

[0063] The disclosed embodiment ensures the timeliness of collected information by collecting the first timestamp and the second timestamp of the information, thereby improving the accuracy of the simulation effect.

[0064] In another example, the method may further include the following operation: determining whether the first timestamp and the third timestamp corresponding to the first position meet a second predetermined condition, and if so, deleting the original acquisition information; if not, determining the original acquisition information as the acquisition information. The second predetermined condition may be, for example, that the interval between the first timestamp and the third timestamp is greater than or equal to a second predetermined time length, and the second predetermined time length may be, for example, 5 milliseconds.

[0065] For example, the third timestamp and the first timestamp when the target object is at the second position can be compared to obtain a second time interval between the two. For example, when the second interval time meets the second predetermined condition, it can be represented that at a certain moment, the target object is located at a specific position in the virtual environment scene, and at this moment or a moment close to this moment, the sensor collects the collected data.

[0066] By limiting the second interval duration, the validity of the data can be ensured. For example, at the first moment, the virtual reference object is at position A in the virtual environment scene, and at the first moment, the sensor's collected data a represents that the distance between the target object and the reference object in the real scene (such as the laboratory above) is 50 meters. Therefore, in the virtual environment scene, a virtual object can be generated at a position 50 meters away from position A. At the second moment, the virtual reference object is at position B in the virtual environment scene, and at the second moment, the sensor's collected data b represents that the distance between the target object and the reference object in the real scene (such as the laboratory above) is 20 meters. Therefore, in the virtual environment scene, a virtual object can be generated at a position 20 meters away from position B. If the above-mentioned second predetermined condition is not considered, a virtual object may be generated at a position 50 meters away from position B at the second moment, thereby causing the problem of inaccurate position of the virtual object in the virtual environment scene.

[0067] The embodiment of the present disclosure ensures the timeliness of the collected information by collecting the first timestamp and the third timestamp of the information, thereby improving the accuracy of the simulation effect.

[0068] In another example, the original collected information may be deleted when the first timestamp and the second timestamp satisfy a first predetermined condition, and / or when the first timestamp and the third timestamp satisfy a second predetermined condition.

[0069] In another example, at least one original acquisition information can be transformed from the sensor coordinate system to the virtual environment scene coordinate system to obtain at least one transformed acquisition information, and then it is determined whether the transformed acquisition information is outside the boundary information. If so, the transformed acquisition information and / or the original acquisition information corresponding to the transformed acquisition information can be deleted; if not, the transformed acquisition information and / or the original acquisition information corresponding to the transformed acquisition information can be determined as the acquisition information.

[0070] For example, taking the original collected information as point cloud data collected by a lidar, after coordinate transformation, the point cloud data outside the virtual environment scene can be filtered out, and the filtered point cloud data no longer participates in the process of determining the perception information.

[0071] By adopting the technical solution provided by the embodiment of the present disclosure, since the transformed collected information exceeds the boundary of the virtual environment scene, it will be impossible to determine the position of the virtual object based on the map data corresponding to the virtual environment scene, resulting in inaccurate position of the virtual object. Therefore, the collected information that exceeds the boundary can be deleted to ensure that the position of the virtual object in the virtual environment scene is accurately reflected. In addition, since a part of the collected information is deleted before using the collected information to determine the perceived information, the amount of calculation for the electronic device to use the collected information to determine the perceived information can be reduced, thereby improving data processing efficiency.

[0072] According to another embodiment of the present disclosure, the above method may further include the following operations: determining actual information for the target object, and then determining the accuracy of the perceived information based on the actual information.

[0073] For example, the actual information may represent the true category, true position, true orientation, and other true value information of the target object.

[0074] For example, the actual information of the target object can be planned in advance, and then the target object can be placed based on the actual information. Alternatively, after placing the target object, another perception algorithm that has been verified can be used to detect the target object, and the perception result output by the perception algorithm can be used as the actual information.

[0075] For example, actual information can be compared with perceived information to determine whether the perceived information is accurate. When an unverified perception algorithm is used to process data to obtain perceived information, the difference between the actual information and the perceived information can be used to determine whether the unverified perception algorithm is accurate, thereby facilitating users to debug the perception algorithm according to actual needs.

[0076] Figure 3 is a schematic diagram of a virtual object simulation method according to an embodiment of the present disclosure.

[0077] like Figure 3 As shown, in this embodiment, a scene selector 302 can be used to select a virtual environment scene 303 from multiple virtual environment scenes 301 as the virtual environment scene 303 to be used currently, and the virtual environment scene 303 corresponds to map data. The sensor 304 can be used to collect information about surrounding target objects. The virtual environment scene 303 can also be displayed using a human-computer interaction interface, and a virtual reference object corresponding to the reference object is generated at a predetermined position in the virtual environment scene 303, and the position of the virtual reference object in the virtual environment scene 303 is obtained, and the position can be used as indoor positioning 305.

[0078] Next, the perception information may be determined based on the virtual environment scene 303, the information collected by the sensor 304, and the indoor positioning 305. For example, the above information is input into the perception algorithm 306 to be verified, and the perception algorithm 306 outputs the perception information.

[0079] Then, a virtual object corresponding to the target object can be generated in the virtual environment scene 303, thereby obtaining a target virtual scene 307. Through the target virtual scene 307, the perception algorithm 306 can be verified.

[0080] Figure 4 It is a schematic structural block diagram of a virtual object simulation device according to an embodiment of the present disclosure.

[0081] like Figure 4As shown, the present disclosure further provides a virtual object simulation device 400 , which includes a sensing mechanism and an electronic device 401 .

[0082] The sensing mechanism may include a rack and at least one sensor. The rack may be placed in a predetermined real scene, such as a laboratory for simulation testing. The sensor is mounted on the rack, and the sensor may include at least one of a laser radar 402, a millimeter wave radar 403, an ultrasonic radar 404, an image acquisition device 405, and an inertial navigation device 406.

[0083] The electronic device 401 can be used to perform the above-mentioned virtual object simulation method. In some embodiments, the electronic device 401 may include multiple software modules, such as a scene selector, a fault switch, an indoor positioning module, etc. In practical applications, the user can pre-build a virtual environment scene according to actual needs, and then use the scene selector to select the virtual environment scene currently required to be used. Then the indoor positioning module uses the virtual environment scene and map data to determine the position of the virtual reference object in the virtual environment scene. The effect of simulating a sensor failure can be achieved through a fault switch. The acquisition information of the sensor can also be obtained, and then the perception information is determined based on the position information of the reference object, the acquisition information of the sensor, the virtual environment scene, the timestamp, etc. When the perception information is determined using the perception algorithm, the effect of verifying the perception algorithm can be achieved, so that the user can debug the perception algorithm according to the verification result.

[0084] In some embodiments, the virtual object simulation device may further include a display device 407 and an input device. The display device 407 may include a display screen for displaying a human-computer interaction interface, and the input device may include a keyboard 408, a mouse 409, and the like.

[0085] In some embodiments, the virtual object simulation device can be placed indoors for use. Physical models such as vehicles, cones, and pedestrians can be placed indoors as target objects. Sensors can be used to collect data about the target objects, and the collected data can be used to generate virtual objects in a virtual environment. Since the virtual object simulation device does not need to be driven on a real road, the device can be used as a teaching product to facilitate students to learn about autonomous driving.

[0086] In related technologies, data of typical scenes can be collected by a collection vehicle equipped with various sensors, and then the perception effect can be verified using the collected data of various real scenes. It is understandable that the cost of adopting this solution is high, and it is restricted by road traffic regulations and has low operational convenience.

[0087] Compared with the above-mentioned technical solutions, the virtual object simulation device provided by the embodiment of the present disclosure can be applied in indoor environments. Users can complete data collection, scene simulation, perception effect hardware-in-the-loop simulation and other processes indoors, thereby improving the research and development efficiency of autonomous driving vehicle perception algorithms, and can also solve the problem of high data collection costs when performing simulations based on collected data.

[0088] Figure 5 It is a schematic structural block diagram of a virtual object simulation device according to an embodiment of the present disclosure.

[0089] like Figure 5 As shown, the virtual object simulation device 500 may include a first determination module 510 , a second determination module 520 , a third determination module 530 and a first generation module 540 .

[0090] The first determining module 510 is used to determine a virtual environment scene.

[0091] The second determination module 520 is used to determine the collected information for the target object, where the collected information is obtained by collecting the target object in a predetermined real scene using a sensor.

[0092] The third determination module 530 is used to determine the perception information for the target object according to the virtual environment scene, the collected information and the position information of the reference object. The position information of the reference object includes: the first position of the reference object in the virtual environment scene coordinate system and the second position of the reference object in the sensor coordinate system.

[0093] The first generating module 540 is used to generate a virtual object corresponding to the target object in the virtual environment scene according to the perception information.

[0094] According to another embodiment of the present disclosure, the second determination module includes a first acquisition submodule, a first determination submodule, a first deletion submodule and a second determination submodule. The first acquisition submodule is used to acquire at least one original acquisition information for at least one target object, each original acquisition information includes a first timestamp indicating the acquisition time. The first determination submodule is used to determine a second timestamp corresponding to each original acquisition information, the second timestamp indicating the time when the original acquisition information is received. The first deletion submodule is used to delete the original acquisition information from at least one original acquisition information in response to determining that the first timestamp, the second timestamp and the third timestamp corresponding to the first position meet a predetermined condition for each original acquisition information. The second determination submodule is used to determine the remaining original acquisition information as the acquisition information.

[0095] According to another embodiment of the present disclosure, the predetermined condition includes at least one of the following: the interval between the first timestamp and the second timestamp is greater than or equal to a first predetermined duration, and the interval between the first timestamp and the third timestamp is greater than or equal to a second predetermined duration.

[0096] According to another embodiment of the present disclosure, the second determination module includes a second acquisition submodule, a transformation submodule, a second deletion submodule and a third determination submodule. The second acquisition submodule is used to acquire at least one original acquisition information for at least one target object. The transformation submodule is used to transform at least one original acquisition information from the sensor coordinate system to the virtual environment scene coordinate system, respectively, to obtain at least one transformed acquisition information. The second deletion submodule is used to delete the transformed acquisition information located outside the boundary information in at least one transformed acquisition information. The third determination submodule is used to determine the acquisition information based on the remaining transformed acquisition information.

[0097] According to another embodiment of the present disclosure, the second determination module includes an opening submodule, a third acquisition submodule and a fourth determination submodule. The opening submodule is used to open the fault switch related to the sensor in response to receiving the opening instruction. The third acquisition submodule is used to obtain the original acquisition information when the fault switch is opened. The fourth determination submodule is used to determine the acquisition information based on the original acquisition information.

[0098] According to another embodiment of the present disclosure, the above-mentioned device further includes a fourth determination module and a fifth determination module. The fourth determination module is used to determine the actual information for the target object. The fifth determination module is used to determine the accuracy of the perception information based on the actual information.

[0099] According to another embodiment of the present disclosure, the apparatus further includes a second generation module and a sixth determination module. The second generation module is used to generate a virtual reference object in the virtual environment scene in response to receiving a generation instruction and according to the generation instruction. The sixth determination module is used to determine the position of the virtual reference object in the virtual environment scene as the first position according to the map data corresponding to the virtual environment scene.

[0100] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0101] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0102] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, comprising at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned virtual object simulation method.

[0103] According to an embodiment of the present disclosure, the present disclosure further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the above-mentioned virtual object simulation method.

[0104] According to an embodiment of the present disclosure, the present disclosure further provides a computer program product, including a computer program, and the computer program implements the above virtual object simulation method when executed by a processor.

[0105] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0106] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0107] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0108] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as a virtual object simulation method. For example, in some embodiments, the virtual object simulation method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the virtual object simulation method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the virtual object simulation method in any other appropriate manner (e.g., by means of firmware).

[0109] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0110] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0111] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0112] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0113] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0114] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0115] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0116] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A virtual object simulation method, comprising: Determine the virtual environment scene; Determining the collection information for the target object includes: for each of the original collection information of the target object, in response to determining that the first timestamp and the second timestamp satisfy a first predetermined condition or the first timestamp and the third timestamp satisfy a second predetermined condition, deleting the original collection information from the at least one original collection information; determining the remaining original collection information as the collection information for the target object; the collection information is collected by using a sensor on the target object in a predetermined real scene; Determining perception information for the target object according to the virtual environment scene, the collected information, and the position information of the reference object; and Generating a virtual object corresponding to the target object in the virtual environment scene according to the perception information; The position information of the reference object includes: a first position of a virtual reference object corresponding to the reference object in a virtual environment scene coordinate system, and a second position of the reference object in a sensor coordinate system; The first timestamp indicates the acquisition time of the original acquisition information, the second timestamp indicates the time when the original acquisition information is received, and the third timestamp indicates the time when the virtual reference object is at the first position.

2. The method according to claim 1, wherein: Determining the collection information for the target object includes: Acquire at least one piece of original collected information for at least one target object; and determine a second timestamp corresponding to each piece of original collected information.

3. The method according to claim 2, wherein: The first predetermined condition is: the interval between the first timestamp and the second timestamp is greater than or equal to a first predetermined time length; The second predetermined condition is that the interval between the first timestamp and the third timestamp is greater than or equal to a second predetermined time length.

4. The method according to claim 1, wherein: Determining the collection information for the target object includes: transforming the at least one original collected information from the sensor coordinate system to the virtual environment scene coordinate system to obtain at least one transformed collected information; and The transformed collected information located outside the boundary information in the at least one transformed collected information is deleted.

5. The method according to claim 1, wherein: Determining the collection information for the target object includes: In response to receiving the opening instruction, opening a fault switch associated with the sensor; When the fault switch is turned on, obtaining original collected information; and The collected information is determined according to the original collected information.

6. The method according to claim 1, further comprising: Determine the actual message for the target audience; as well as The accuracy of the perceived information is determined based on the actual information.

7. The method according to claim 1, further comprising: In response to receiving a generation instruction, generating a virtual reference object in the virtual environment scene according to the generation instruction; as well as According to the map data corresponding to the virtual environment scene, the position of the virtual reference object in the virtual environment scene is determined as the first position.

8. A virtual object simulation device, comprising: A first determination module is used to determine a virtual environment scene; A second determination module is used to determine the collected information for the target object, where the collected information is collected from the target object in a predetermined real scene using a sensor; A third determination module is used to determine the perception information for the target object according to the virtual environment scene, the collected information and the position information of the reference object; as well as A first generating module, configured to generate a virtual object corresponding to the target object in a virtual environment scene according to the perception information; Wherein, the second determining module includes: a first deletion submodule, configured to, for each of the at least one original collection information of the target object, delete the original collection information from the at least one original collection information in response to determining that the first timestamp and the second timestamp satisfy a first predetermined condition or the first timestamp and the third timestamp satisfy a second predetermined condition; and A second determination submodule is used to determine the remaining original collected information as collected information for the target object; The position information of the reference object includes: a first position of a virtual reference object corresponding to the reference object in a virtual environment scene coordinate system, and a second position of the reference object in a sensor coordinate system; The first timestamp indicates the acquisition time of the original acquisition information, the second timestamp indicates the time when the original acquisition information is received, and the third timestamp indicates the time when the virtual reference object is at the first position.

9. The device according to claim 8, wherein: The second determining module further includes: A first acquisition submodule is used to acquire at least one piece of original collected information for at least one target object; The first determining submodule is used to determine a second timestamp corresponding to each piece of original collected information.

10. The device according to claim 9, wherein: The first predetermined condition is: the interval between the first timestamp and the second timestamp is greater than or equal to a first predetermined time length; The second predetermined condition is that the interval between the first timestamp and the third timestamp is greater than or equal to a second predetermined time length.

11. The device according to claim 8, wherein: The second determining module further includes: a transformation submodule, configured to transform the at least one original collected information from the sensor coordinate system to the virtual environment scene coordinate system to obtain at least one transformed collected information; and The second deleting submodule is used to delete the transformed collected information located outside the boundary information in the at least one transformed collected information.

12. The device according to claim 8, wherein: The second determining module comprises: an opening submodule, configured to open a fault switch associated with the sensor in response to receiving an opening instruction; A third acquisition submodule is used to acquire original acquisition information when the fault switch is turned on; and The fourth determining submodule is used to determine the collected information according to the original collected information.

13. The apparatus according to claim 8, further comprising: A fourth determination module, used to determine actual information for the target object; as well as The fifth determination module is used to determine the accuracy of the perception information based on the actual information.

14. The apparatus according to claim 8, further comprising: A second generating module, configured to generate a virtual reference object in the virtual environment scene in response to receiving a generating instruction and according to the generating instruction; as well as The sixth determination module is used to determine the position of the virtual reference object in the virtual environment scene according to the map data corresponding to the virtual environment scene as the first position.

15. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

17. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

18. A virtual object simulation device, comprising: The electronic device as claimed in claim 15; as well as The sensing mechanism comprises a frame and at least one sensor installed on the frame, each sensor being used to detect a target object located around the frame to obtain collected information.

19. The virtual object simulation device according to claim 18, wherein: The at least one sensor includes at least one of the following: a laser radar, a millimeter wave radar, an ultrasonic radar, an image acquisition device, and an inertial navigation device.

Images