Occlusion judgment method and related device, rejection method, module and simulation platform

Abstract

This invention pertains to an occlusion calculation and judgment method in a simulation platform. To address the technical problems of current occlusion calculation and judgment methods used in autonomous driving simulation platforms developed based on Unreal Engine, such as the inability to remove dynamic objects, high CPU computational consumption, reliance on GPUs, and the need for depth information, this invention provides an occlusion judgment method and related devices, removal methods, modules, and a simulation platform. First, during the simulation process, multiple rays are continuously emitted outward through pre-positioned sensors located in the simulated world. Then, the number of times each object hit by a ray is recorded is obtained. Finally, for each object hit by a ray, the ratio of the number of hits to the theoretical number of hits is used to determine whether the object is occluded. This significantly reduces computational complexity, eliminates reliance on GPU hardware resources and the use of depth information, has a short processing time, high judgment accuracy, and can simultaneously handle both dynamic and static objects.

Description

Technical Field

[0001] This invention relates to an occlusion calculation and judgment method in a simulation platform, specifically to an occlusion judgment method and related devices, removal methods, modules, and simulation platform. Background Technology

[0002] In computer graphics applications, such as game applications, occlusion culling is a very important technique that can help identify objects that do not need to be rendered, thereby reducing overhead.

[0003] Currently, autonomous driving simulation platforms developed based on game engines generally use the following methods for occlusion calculation and judgment:

[0004] (1) Pre-calculate visibility. First, divide the scene into multiple cells, then pre-calculate all objects that the camera may see within the area of ​​each cell, and save the information. In subsequent runtime, the visibility information of all static objects can be obtained directly by looking up a table.

[0005] (2) Software-culling. Large objects used for occlusion are manually marked. During runtime, the bounding boxes of the occluded objects are soft rasterized onto the Z-Buffer in the CPU memory. Then, based on the depth information in the Z-Buffer, the bounding boxes of the objects are culled as needed, and the occlusion information is calculated in real time.

[0006] (3) Occlusion query. First, use a depth-only pass to write the depth into the Z-Buffer. Then, use the bounding box of the object to pass to the GPU for occlusion testing. If the test finds that all pixels are occluded, it means that the object is occluded; otherwise, it is considered visible.

[0007] (4) HZB (Hierarchical Z-Buffer). HZB is a multi-level Z-Buffer, where each higher-level buffer records the depth value of the farthest of the four points surrounding the previous level. After HZB is generated, the bounding box information of the object to be culled can be passed to the Computer Shader for calculation. During calculation, the most suitable level is selected for occlusion testing. Objects occupying a larger area on the screen will be tested at a higher depth level, which can reduce the amount of computation.

[0008] However, the above method still has the following drawbacks:

[0009] (1) When pre-compiling visibility, dynamic objects cannot be eliminated, and the visibility of static objects is immutable;

[0010] (2) Software culling consumes a lot of computation on the CPU side and requires depth information;

[0011] (3) Occlusion queries and HZB rely on GPU and also require depth information. Summary of the Invention

[0012] To address the technical problems of current occlusion calculation and judgment methods used in autonomous driving simulation platforms developed based on Unreal Engine, such as the inability to remove dynamic objects, high CPU-side computational consumption, reliance on GPU, and the need for depth information, this invention provides an occlusion judgment method and related devices, removal methods, modules, and simulation platforms.

[0013] To achieve the above objectives, the present invention employs the following technical solution:

[0014] In a first aspect, the present invention provides an occlusion determination method, comprising the following steps:

[0015] During the simulation, multiple rays are continuously emitted outward through pre-placed sensors located in the simulated world;

[0016] Get the number of times each object hit by the ray has been hit;

[0017] For each object hit by the ray, it is determined whether the object is blocked based on the ratio of the number of times it is hit to the theoretical number of times it is hit; the theoretical number of times it is hit is determined in conjunction with the number of rays.

[0018] Preferably, the theoretical number of hits is determined by combining the number of rays, the impact surface where the object hit by the ray is located, and the projection of the object's surrounding box onto the impact surface.

[0019] The impact surface is a plane whose normal vector is the direction of the ray emission that hits the object struck by the ray.

[0020] Preferably, the theoretical number of hits is obtained by the following formula:

[0021]

[0022] Where A is the theoretical number of hits, P is the projected area of ​​the bounding box of the object hit by the ray on the impact surface, S is the area of ​​the impact surface where the object hit by the ray is located, and N is the number of rays.

[0023] Preferably, determining whether an object hit by radiation is blocked based on the ratio of the number of times it is hit to the theoretical number of times it is hit specifically involves:

[0024] The proportion B of the object hit by the ray is blocked by the following formula:

[0025]

[0026] Where H is the number of times the object hit by the ray is hit;

[0027] If the occlusion ratio B of the object hit by the ray is greater than the preset value, it is determined that the object hit by the ray is occluded; otherwise, it is determined that the object hit by the ray is not occluded.

[0028] Secondly, the present invention provides an occlusion determination system, comprising a ray module, a statistics module, and a determination module:

[0029] The ray module is used to continuously emit multiple rays outward through a pre-set sensor located in the simulation world during the simulation process;

[0030] The statistics module is used to obtain the number of times each object hit by the ray was hit.

[0031] The judgment module is used to determine whether an object hit by a ray is blocked based on the ratio of the number of times it is hit to the theoretical number of times it is hit; the theoretical number of times it is hit is determined in conjunction with the number of rays.

[0032] Thirdly, the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0033] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements the steps of the above-described method.

[0034] Fifthly, the present invention provides an occlusion removal method, comprising the following steps:

[0035] Using the above-mentioned occlusion judgment method, if the object hit by the ray is occluded, the object hit by the ray is discarded; otherwise, the object hit by the ray is retained.

[0036] In a sixth aspect, the present invention provides an occlusion removal module, which, when calculating the target information after occlusion removal, uses the above-mentioned occlusion removal method to remove or retain objects hit by rays.

[0037] In a seventh aspect, the present invention provides a simulation platform, including Unreal Engine, a server and a client, characterized in that: the server is provided with the aforementioned occlusion culling module.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] The occlusion determination method of this invention converts area calculation into a proportional calculation of ray emission count when determining whether occlusion has occurred. This significantly reduces computational complexity, eliminating reliance on GPU hardware resources and depth information. It boasts short processing time, high accuracy, minimal hardware resource requirements, and does not consume the computing power of the simulation platform. Furthermore, it can handle both dynamic and static objects simultaneously. In addition, it allows for real-time determination, offering high real-time performance suitable for simulation platform environments without affecting the overall simulation test time delay.

[0040] Furthermore, in calculating the theoretical number of hits, this invention combines the number of rays, the area of ​​the impact surface where the object hit by the ray is located, and the projected area of ​​the enclosing box of the object hit by the ray on the impact surface. The calculation method is simple and easy to implement.

[0041] The occlusion judgment system of the present invention can implement the above-mentioned occlusion judgment method in a modular form and promote its application.

[0042] The occlusion removal method of the present invention makes a judgment based on the above-mentioned occlusion judgment method. If the object hit by the ray is occluded, the object hit by the ray is removed; otherwise, the object hit by the ray is retained. This method can optimize the scene in the virtual world, making it closer to the real world and improving the user experience.

[0043] The simulation platform of this invention enables the occlusion removal module in the server to calculate the target information after occlusion removal using the above-mentioned occlusion removal method, thereby improving the effect of the target detection sensor of the simulation platform and the performance of downstream functional modules, and effectively enhancing the overall competitiveness of the simulation platform. Attached Figure Description

[0044] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a structural framework diagram of the Carla autonomous driving simulation platform;

[0046] Figure 2 This is a flowchart illustrating Embodiment 1 of the present invention;

[0047] Figure 3 This is a flowchart illustrating Embodiment 2 of the present invention;

[0048] Figure 4This is a schematic diagram showing the projected area of ​​the enclosure of the object struck by the ray on the impact surface and the area of ​​the impact surface where the object is located, in Embodiment 2 of the present invention.

[0049] Figure 5 This is a schematic diagram of Embodiment 3 of the present invention;

[0050] Figure 6 This is a schematic diagram of Embodiment 4 of the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0052] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0053] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0054] In the description of the embodiments of the present invention, it should be noted that if terms such as "upper," "lower," "horizontal," or "inner" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of the invention is in use, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. Furthermore, terms such as "first" and "second" are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0055] Furthermore, the use of the term "horizontal" does not imply that the component must be absolutely horizontal, but rather that it can be slightly tilted. For example, "horizontal" simply means that its direction is more horizontal than "vertical," and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0056] In the description of the embodiments of the present invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention according to the specific circumstances.

[0057] In the field of autonomous driving simulation testing, simulation software can provide users with complete information on all elements of the simulated world, including their coordinates, orientation, and size. Similar to real sensors, when users conduct tests using virtual sensors in the simulated world, the virtual sensors also have limitations in imaging distance and field of view, serving as a one-to-one clone of real-world sensors. Likewise, objects occluded in a real camera's view should not be displayed in a virtual camera.

[0058] like Figure 1 The diagram shows the structural framework of Carla, a typical autonomous driving simulation platform. Based on Unreal Engine, it provides scene rendering capabilities and uses a server-client mechanism established through network communication and calling protocols to simulate the 3D world. It mainly consists of a server and a client. The server is responsible for the simulation itself and can update the virtual world based on external commands from the client. The server contains various sensor models simulating the real world, including cameras, LiDAR, acoustic radar, IMU, GNSS, etc. How the virtual world should operate at different times is controlled by the client. Users can write Python scripts to send commands to the server, guiding changes in the virtual world, which the server then executes. Simultaneously, the client can also receive information from the server, such as photos taken by a camera in the virtual world.

[0059] From a data flow perspective, Unreal Engine uses the GPU to complete the rendering pipeline calculation process. After the simulation platform obtains the rendering results through a plugin, it sends them to the user end via a network communication protocol. In the Carla framework, the sensor established on the simulation platform server calls the ray emission function to obtain the ray hit results calculated by the underlying Unreal Engine. After the hit information is obtained through function calls, it is processed by the occlusion culling module on the server to calculate the target information after occlusion culling. Then, the final calculated target detection information is sent to the test user end through the communication module for downstream testing, prediction, planning, and control.

[0060] To ensure high fidelity in simulation testing, elements need to be filtered based on the actual field of view of the sensor. Occlusion culling is performed on the imaging of elements in the virtual world. This means that, with the same parameter configuration as a real camera, the virtual camera also has an actual visible range. Objects outside this visible range need to be removed. This is a crucial object detection task in autonomous driving perception testing: identifying the bounding boxes of objects. The generation of bounding boxes in Unreal Engine is directly related to the skeleton of the 3D model of the elements. Unreal Engine's built-in occlusion culling cannot filter the skeleton, resulting in the bounding boxes of occluded objects still being displayed in the simulation environment. This invention addresses this problem by proposing an occlusion culling method.

[0061] The following are embodiments of the present invention, used to provide a detailed description of the present invention:

[0062] Example 1

[0063] like Figure 2 The diagram shows a basic embodiment of an occlusion culling method according to the present invention, and the specific steps are as follows:

[0064] S101, during the simulation, continuously emits multiple rays outward through a pre-set sensor located in the simulated world. The pre-set sensor, positioned in the simulated world, is a realistic sensor capable of emitting rays outward.

[0065] S102, Get the number of times all objects hit by the ray were hit.

[0066] S103, for each object hit by the ray, determine whether the object is blocked based on the ratio of the number of times it was hit to the theoretical number of times it was hit. Furthermore, the theoretical number of times it was hit in this invention is determined in conjunction with the number of rays.

[0067] Example 2

[0068] like Figure 3 The image shows a specific embodiment of an occlusion removal method according to the present invention, and the specific steps are as follows:

[0069] S201, in the simulated world, sets up a pre-set sensor that can emit rays.

[0070] S202, at each moment during the simulation, emits a certain number of rays within a specific field of view through a preset sensor. The range of the field of view and the number of rays can be adjusted according to the conditions of the simulated world. For example, a higher number of rays results in higher calculation accuracy, but the difficulty of setting up the preset sensor and the settings of the simulated virtual world must also be considered.

[0071] S203: Count the hits of all emitted rays. If a ray hits an object, record the identifier of the object hit by the ray, and finally obtain the hit count H for each object hit by a ray. Regarding the object identifier, since the simulated virtual world is a constructed virtual world, all objects in the simulated virtual world have an identifier used to distinguish them from other objects during the construction process. Recording this identifier means recording the object as an object hit by a ray, and thus obtaining the hit count of all objects hit by rays.

[0072] S204, determine the impact surface and the area of ​​the impact surface.

[0073] like Figure 4 For each object hit by a ray, when the ray is emitted, the location of the preset sensor is taken as the initial point, and all rays are emitted from the initial point, forming a cone-like area. The distance between the object hit by the ray and the sensor is obtained when the object is hit. Then, the ray emission direction that hits the object is taken as the normal vector, and a cross section perpendicular to the axis of the cone can be determined. This cross section is denoted as the impact surface, and the area of ​​this impact surface is S. Figure 4 In the example, C represents the object being hit. In the second example, a car is used as an example. D represents the surrounding box of the object being hit.

[0074] S205, obtain the projected area of ​​the bounding box of the object hit by the ray on the impact surface. Then, the theoretical number of times the object hit by the ray is hit without obstruction is:

[0075]

[0076] Where A is the theoretical number of hits, P is the projected area of ​​the bounding box of the object hit by the ray on the impact surface, S is the area of ​​the impact surface where the object hit by the ray is located, and N is the number of rays.

[0077] S206, the final occlusion ratio B of the object hit by the ray can be calculated using the following formula:

[0078]

[0079] If the occlusion ratio B of the object hit by the ray is greater than the preset value, it is determined that the object hit by the ray is occluded; otherwise, it is determined that the object hit by the ray is not occluded.

[0080] The preset values ​​here can be set according to the required judgment criteria.

[0081] This invention first emits rays within a certain viewing angle range to obtain ray hit information, then counts the number of times each visible object is hit, and then calculates the theoretical number of hits by using the ratio of the projected area of ​​the object's bounding box on the ray normal vector plane to the viewing angle cross-sectional area at the current position. Finally, using the actual number of hits and the theoretical number of hits, the occlusion ratio of the object is calculated.

[0082] Example 3

[0083] like Figure 5 As shown, in order to implement the above-mentioned occlusion judgment method, the present invention also proposes an occlusion judgment system, including a ray module, a statistics module and a judgment module.

[0084] The ray module is used to continuously emit multiple rays outward through a pre-set sensor located in the simulation world during the simulation process;

[0085] The statistics module is used to obtain the number of times each object hit by the ray was hit.

[0086] The judgment module is used to determine whether an object hit by a ray is blocked based on the ratio of the number of times it is hit to the theoretical number of times it is hit; the theoretical number of times it is hit is determined in conjunction with the number of rays.

[0087] As a preferred embodiment of the third embodiment, the ray module, the statistics module, and the judgment module can be specifically implemented according to the preferred embodiment of the second embodiment.

[0088] In other embodiments of the present invention, the ray module, the statistics module, and the judgment module can be combined accordingly or further divided, with each sub-function in each module being implemented through more sub-modules.

[0089] A computer device is provided according to an embodiment of the present invention. This computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps in the various method embodiments described above. Alternatively, when the processor executes the computer program, it implements the functions of each module / unit in the various device embodiments described above.

[0090] The computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention.

[0091] The computer device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device may include, but is not limited to, a processor and memory.

[0092] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0093] The memory can be used to store the computer program and / or module, and the processor implements various functions of the computer device by running or executing the computer program and / or module stored in the memory, and by calling the data stored in the memory.

[0094] If the modules / units integrated into the computer device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0095] Example 4

[0096] like Figure 6 As shown, based on the above occlusion judgment method, this invention also proposes an occlusion removal method:

[0097] S301, the object hit by the ray is determined to be whether it is blocked by any embodiment of the aforementioned occlusion determination method;

[0098] S302: If the object hit by the ray is blocked, then discard the object hit by the ray; otherwise, retain the object hit by the ray.

[0099] In simulation, the occlusion culling function improves the accuracy and rationality of the output results of the target detection sensor on the simulation test platform. Only objects whose occlusion ratio is less than the set threshold will be displayed. In the autonomous driving test process, the target detection results (position and size of the bounding box) are an important output of the perception module in the test chain. They are the premise and dependence of downstream path planning, motion prediction, collision avoidance and other functions. The occlusion culling function optimizes the output results of the target detection sensor and improves the overall performance of autonomous driving simulation test.

[0100] Accordingly, by adopting the occlusion removal method of the present invention, an occlusion removal module can be obtained. When calculating the target information after occlusion removal, the occlusion removal module uses the above-mentioned occlusion removal method to remove or retain the object hit by the ray.

[0101] The occlusion culling module of the present invention is used in a simulation platform to obtain a simulation platform proposed in the present invention, which includes Unreal Engine, server and client, wherein the server is equipped with the occlusion culling module proposed in the present invention.

[0102] This invention utilizes the concept of ray emission and combines it with solid geometry knowledge. The calculation process is relatively simple, with minimal processing time, ensuring real-time performance. Deployment in the product does not affect the overall simulation test latency, it is not GPU-dependent, requires almost no hardware resources, and does not consume the simulation platform's computing power. It can simultaneously handle dynamic and static objects, satisfying most testing scenarios. In summary, this invention, based on real-time performance, hardware independence, and dynamic / static compatibility, improves the effectiveness of the target detection sensor in the simulation platform and the performance of downstream functional modules, enhancing the product's competitiveness.

[0103] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An occlusion determination method, characterized in that, Includes the following steps: During the simulation, multiple rays are continuously emitted outward through pre-placed sensors located in the simulated world; Get the number of times each object hit by the ray has been hit; For each object hit by the ray, determine whether the object is obstructed based on the ratio of the number of times it is hit to the theoretical number of times it is hit: calculate the obstruction ratio of the object hit by the ray using the following formula. : in, This represents the number of times an object hit by a ray was struck. The theoretical number of hits; If the proportion of the object hit by the ray is blocked If the value is greater than the preset value, it is determined that the object hit by the ray is blocked; otherwise, it is determined that the object hit by the ray is not blocked. The theoretical number of hits is determined in combination with the number of rays. The theoretical number of hits is obtained by the following formula: in, The theoretical number of hits. Let be the projected area of ​​the bounding box of the object hit by the ray on the impact surface. Let be the area of ​​the surface of the object struck by the ray. The number of rays is denoted as ; the impact surface is a plane whose normal vector is the direction in which the rays emitted from the object being hit by the rays strike.

2. An occlusion detection system, characterized in that: Includes a ray casting module, a statistics module, and a judgment module: The ray module is used to continuously emit multiple rays outward through a pre-set sensor located in the simulation world during the simulation process; The statistics module is used to obtain the number of times each object hit by the ray was hit. The judgment module is used to determine, for each object hit by the ray, whether the object is obstructed based on the ratio of the number of times it is hit to the theoretical number of times it is hit: the obstruction ratio of the object hit by the ray is calculated using the following formula. : in, This represents the number of times an object hit by a ray was struck. The theoretical number of hits; If the proportion of the object hit by the ray is blocked If the value is greater than the preset value, it is determined that the object hit by the ray is blocked; otherwise, it is determined that the object hit by the ray is not blocked. The theoretical number of hits is determined in combination with the number of rays. The theoretical number of hits is obtained by the following formula: in, The theoretical number of hits. Let be the projected area of ​​the bounding box of the object hit by the ray on the impact surface. Let be the area of ​​the surface of the object struck by the ray. The number of rays is denoted as ; the impact surface is a plane whose normal vector is the direction in which the rays emitted from the object being hit by the rays strike.

3. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in claim 1.

4. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in claim 1.

5. An occlusion removal method, characterized in that, Includes the following steps: The occlusion determination method described in claim 1 is used for determination. If the object hit by the ray is occluded, the object hit by the ray is discarded; otherwise, the object hit by the ray is retained.

6. An occlusion removal module, characterized in that: When calculating the target information after occlusion removal, the occlusion removal module uses the occlusion removal method described in claim 5 to remove or retain objects hit by rays.

7. A simulation platform, comprising Unreal Engine, a server, and a client, characterized in that: The server is equipped with the occlusion removal module as described in claim 6.

Images