Remote control driving safety control method and device, medium and vehicle

Abstract

The invention relates to the technical field of remote control driving safety control, and particularly discloses a remote control driving safety control method and device, a medium and a vehicle. The method comprises the following steps: firstly, fusing three types of sensor data acquired by an image, a first point cloud device and a second point cloud device, and carrying out time-space synchronization and coordinate system unification on the three types of sensor data; then, whether a medium causing interference on the detection capability of the first point cloud acquisition equipment exists or not is identified based on the image, an untrusted area is divided from the first point cloud data according to the medium, and second point cloud data with better revealing performance is preferentially used in the area to carry out obstacle detection; in other areas, first point cloud data with higher precision is preferentially used for detection; and finally, integrating the two detection results to determine the position and the form of the obstacle so as to generate a vehicle motion control instruction, thereby improving the sensing reliability and the driving safety in complex fire scenes such as dense smoke.

Description

Technical Field

[0001] This application relates to the field of remote driving safety control technology, specifically to a remote driving safety control method, device, medium, and vehicle. Background Technology

[0002] In fire rescue missions, the environment is extremely complex, with dense smoke, high temperatures, and low visibility posing serious threats to the lives of firefighters. To reduce the risk of casualties and improve rescue efficiency, developing remote-controlled driving technology for use in fire scenes has become an important direction. By remotely controlling or semi-autonomous driving fire trucks, firefighting and reconnaissance tasks can be performed in dangerous areas inaccessible to personnel, thus achieving a safe operating mode of "separation of man and machine, remote control."

[0003] However, existing remote-controlled driving systems generally rely on a single sensor, which is difficult to adapt to complex scenes or conditions. For example, using visible light cameras or lidar for environmental perception severely limits performance in harsh conditions such as dense smoke: visible light cameras are obscured by smoke and cannot form images, lidar point clouds are easily interfered with by smoke scattering; and traditional millimeter-wave radar point clouds are sparse and lack precision, failing to accurately identify the location and shape of obstacles. Although some systems are equipped with multiple sensors, the lack of a data fusion mechanism means that the data from each sensor is not aligned in time and space, nor is the trust weight dynamically adjusted according to the scene, resulting in low reliability of obstacle detection and poor system adaptability. Therefore, there is an urgent need for a remote-controlled driving safety control technology that can integrate the advantages of multiple sensor sources and adapt to complex fire scene environments to achieve all-weather, all-scenario obstacle detection capabilities for fire trucks and improve the safety of remote-controlled driving in fire scenes. Summary of the Invention

[0004] The purpose of this application is to provide a remote driving safety control method, device, medium, and vehicle to solve the problems in the prior art where insufficient obstacle detection capability and poor scene adaptability in scenarios with limited field of vision make it difficult for the vehicle's anti-collision detection mechanism to function effectively.

[0005] To achieve the above objectives, the first aspect of this application provides a remote-controlled driving safety control method, comprising: Acquire environmental image data, first point cloud data, and second point cloud data collected by the image acquisition device, the first point cloud acquisition device, and the second point cloud acquisition device, respectively; Data from each acquisition device is synchronized in time and space and processed to unify the coordinate system, so as to associate data from different sources under the same time and space reference. Based on environmental image data, identify whether there are media that interfere with the detection capabilities of the first point cloud acquisition device; Based on the spatial distribution information of the medium in the environmental image data, the untrusted point cloud region corresponding to the medium is distinguished from the first point cloud data; Within the spatial range corresponding to the untrusted point cloud region, obstacle detection is prioritized based on the second point cloud data to obtain the first detection result; Within the spatial range excluding untrusted point cloud areas, obstacle detection is prioritized based on the first point cloud data to obtain the second detection result; The corresponding vehicle motion control command is generated by combining the first and second test results.

[0006] In this embodiment, data spatiotemporal synchronization and coordinate system unification processing are performed on each acquisition device to associate data from different sources under the same spatiotemporal reference. This includes: synchronizing the clock references of the image acquisition device, the first point cloud acquisition device, and the second point cloud acquisition device based on an external clock source; calculating feature vectors for multiple spatial feature points based on the first and second point cloud data to determine the spatial transformation matrix from the coordinate system of the second point cloud data to the coordinate system of the first point cloud data; downsampling the first point cloud data through pass-through filtering and voxel filtering to determine the processed first point cloud data; and projecting the second point cloud data and the processed first point cloud data onto the coordinate system of the environmental image data according to the spatial transformation matrix.

[0007] In this embodiment, based on first point cloud data and second point cloud data, multiple spatial feature points are selected to calculate feature vectors, and the spatial transformation matrix from the coordinate system of the second point cloud data to the coordinate system of the first point cloud data is determined. This includes: selecting at least three non-collinear spatial feature points in the first point cloud data, constructing a local coordinate system and calculating its corresponding three orthogonal feature vectors; identifying corresponding feature points in the second point cloud data that correspond to the spatial positions of the feature points in the first point cloud data, constructing a local coordinate system based on the corresponding feature points, and calculating the corresponding three orthogonal feature vectors; and solving the rotation matrix and translation vector that align the two coordinate systems according to the correspondence between the two sets of feature vectors, thereby obtaining the spatial transformation matrix that transforms the coordinate system of the second point cloud data to the coordinate system of the first point cloud data.

[0008] In this embodiment of the application, identifying whether there is a medium that interferes with the detection capability of the first point cloud acquisition device based on environmental image data includes: detecting the environmental image data based on a pre-trained smoke recognition neural network; and determining that there is a medium that interferes with the detection capability of the first point cloud acquisition device when the smoke confidence level output by the neural network exceeds a preset threshold.

[0009] In this embodiment of the application, distinguishing the untrusted point cloud region corresponding to the medium from the first point cloud data based on the spatial distribution information of the medium in the environmental image data includes: when it is determined that there is a medium that interferes with the detection capability of the first point cloud acquisition device, determining the boundary of the smoke point cloud based on the echo intensity in the first point cloud data; calculating the spatial location and range of the smoke region based on the boundary of the smoke point cloud, and determining the minimum bounding box of the smoke region; and taking the minimum bounding box of the smoke region as the untrusted point cloud region.

[0010] In this embodiment of the application, within the spatial range corresponding to the untrusted point cloud region, obstacle detection is preferentially performed based on the second point cloud data to obtain the first detection result, which includes: acquiring the second point cloud data at the current moment and the previous historical moment; transforming the second point cloud data at the historical moment to the coordinate system at the current moment based on the vehicle pose information to form a multi-frame fused point cloud; extracting obstacle point cloud data of obstacles behind the smoke; performing clustering processing on the multi-frame fused point cloud based on the multi-frame joint clustering method, calculating the position, orientation and boundary information of the obstacle, and combining the changes in the vehicle's position and heading angle at the current moment and the previous historical moment to project the obstacle point cloud data in the vehicle coordinate system to generate the first detection result.

[0011] In this embodiment of the application, within the spatial range excluding the untrusted point cloud region, obstacle detection is preferentially performed based on the first point cloud data to obtain the second detection result, which includes: determining the coordinates of the center point and boundary point of the obstacle based on the first point cloud data; and determining the second detection result based on the coordinates of the center point and boundary point.

[0012] In this embodiment of the application, generating the corresponding vehicle motion control command by combining the first detection result and the second detection result includes: determining the obstacle height based on the first detection result and the second detection result; determining the distance between the vehicle and the obstacle when the obstacle height is higher than a preset height threshold; and generating the vehicle motion control command by combining the vehicle braking state and the vehicle speed state.

[0013] A second aspect of this application provides a remote-controlled driving safety control device, comprising: The memory is configured to store instructions; The processor is configured to retrieve instructions from memory and, when executing the instructions, to implement either of the remote driving safety control methods.

[0014] A third aspect of this application provides a machine-readable storage medium storing instructions for causing a machine to perform any of the remote-controlled driving safety control methods.

[0015] A fourth aspect of this application provides a remotely driven vehicle, comprising: Remote driving safety control device, used to execute remote driving safety control methods; The controller is used to control the vehicle speed.

[0016] Through the above technical solution, this application integrates multi-source sensing data acquired by image acquisition equipment, a first point cloud acquisition equipment, and a second point cloud acquisition equipment, and then performs spatiotemporal synchronization and coordinate system unification to associate the three types of heterogeneous data under the same reference. Furthermore, this application identifies the presence of media interfering with lidar detection based on environmental image data, and determines untrusted point cloud areas accordingly. Within untrusted point cloud areas, it prioritizes obstacle detection using millimeter-wave radar with strong penetration, while in other areas, it prioritizes using higher-precision lidar data. Finally, it fuses the two types of detection results to generate vehicle motion control commands. This technical solution effectively solves the problem of single sensor failure or reliability degradation in complex environments such as dense smoke in fire scenes, significantly improving the obstacle perception accuracy and driving safety of remote-controlled fire trucks in extreme scenarios.

[0017] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description

[0018] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings: Figure 1 The illustration shows a flowchart of a remote driving safety control method according to an embodiment of this application; Figure 2 This schematic diagram illustrates the control logic of a remote-controlled driving safety control method according to an embodiment of this application. Figure 3 This illustration schematically shows a control architecture diagram of a remote driving safety control method according to an embodiment of this application; Figure 4 This schematic diagram illustrates a structural block diagram of a remote-controlled driving safety control device according to an embodiment of this application; Figure 5 The illustration shows a schematic diagram of a remote-controlled vehicle architecture according to an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0020] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of this application all comply with relevant laws and regulations. In the embodiments of this application, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0021] It should be noted that if the embodiments of this application involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0022] Furthermore, if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0023] Figure 1 The illustration schematically shows a flowchart of a remote-controlled driving safety control method according to an embodiment of this application. Figure 1 As shown in the figure, this application provides a remote driving safety control method, which may include the following steps.

[0024] Step 110: Acquire environmental image data, first point cloud data, and second point cloud data collected by the image acquisition device, the first point cloud acquisition device, and the second point cloud acquisition device, respectively.

[0025] In this embodiment, the processor acquires three different types of data collected by three different acquisition devices based on different principles, in order to reduce the impact and limitations that a single type of sensor may encounter in complex environments.

[0026] In one preferred embodiment of this application, a visible light camera is configured as an image acquisition device to acquire high-resolution image data of the fire scene in real time; a lidar is configured as a first point cloud acquisition device to acquire high-density, high-precision three-dimensional point cloud data containing spatial coordinates and echo intensity information as the first point cloud data; and a 4D millimeter-wave radar with good smoke penetration capability but low point cloud density is configured as a second point cloud acquisition device to acquire second point cloud data containing spatial position and radial velocity information.

[0027] Step 120: Perform spatiotemporal synchronization and coordinate system unification processing on each acquisition device to associate data from different sources under the same spatiotemporal reference.

[0028] In this embodiment, the processor synchronizes the clock references of the image acquisition device, the first point cloud acquisition device, and the second point cloud acquisition device based on an external clock source. Then, based on the first point cloud data and the second point cloud data, the processor selects multiple spatial feature points to calculate feature vectors and determines the spatial transformation matrix from the coordinate system of the second point cloud data to the coordinate system of the first point cloud data. Further, the processor downsamples the first point cloud data through pass-through filtering and voxel filtering to determine the processed first point cloud data. Finally, the processor projects the second point cloud data and the processed first point cloud data onto the coordinate system of the environmental image data according to the spatial transformation matrix.

[0029] Specifically, in this embodiment, the processor selects at least three non-collinear spatial feature points in the first point cloud data, constructs a local coordinate system, and calculates the corresponding three orthogonal eigenvectors. In the second point cloud data, the processor identifies the same-name feature points corresponding to the spatial positions of the feature points in the first point cloud data, constructs a local coordinate system based on the same-name feature points, and calculates the corresponding three orthogonal eigenvectors. Finally, the processor solves for the rotation matrix and translation vector that align the two coordinate systems according to the correspondence between the two sets of eigenvectors, thereby obtaining the spatial transformation matrix that transforms the coordinate system of the second point cloud data to the coordinate system of the first point cloud data.

[0030] In this embodiment, the external clock source can be a high-precision clock signal from GNSS (Global Navigation Satellite System). It is understood that GNSS satellites carry high-precision atomic clocks capable of broadcasting extremely accurate time information. By receiving the high-precision clock signal from GNSS and then connecting multiple sensors via serial ports to synchronize the clock source at the hardware level, motion blur or positional offset problems caused by inconsistent sampling times can be effectively eliminated, which is especially crucial in scenarios involving high-speed vehicle movement or dynamic changes in fire conditions.

[0031] In one embodiment of this application, if three non-collinear spatial feature points are selected, the processor can calculate three feature vectors e1, e2, and e3 respectively, which can be represented in the first feature point cloud as follows: , , In the second feature point cloud, it can be represented as , , The formula for transforming the feature vectors in the first point cloud data into the feature vectors in the second point cloud data is:

[0032] Therefore, the rotation matrix from the second point cloud data coordinate system to the first point cloud data coordinate system can be calculated. R for:

[0033] Furthermore, in the embodiments of this application, any set of corresponding feature points can be selected. p If the first feature point cloud is In the second feature point cloud, The translation vector can be obtained. t for:

[0034] Therefore, the spatial transformation matrix from the second point cloud data coordinate system to the first point cloud data coordinate system. T for: T =

[0035] In addition, in this embodiment, a pass-through filter is applied to the lidar point cloud to remove invalid or excessively distant points, while a voxel grid filter reduces the point cloud density while preserving the geometric structure. This improves the efficiency of subsequent processing and avoids computational redundancy caused by excessively dense point clouds.

[0036] Preferably, in this embodiment of the application, after the processor uses an algorithm to filter out the ground point cloud, it uses a clustering algorithm to extract the point cloud of the same obstacle from the first point cloud data and the second point cloud data.

[0037] Ultimately, in this embodiment, the processor can freely project the second point cloud data onto the coordinate system where the first point cloud data is located using the above method, providing a data foundation for subsequent multi-sensor data fusion analysis and obstacle recognition.

[0038] Step 130: Based on the environmental image data, identify whether there is a medium that interferes with the detection capability of the first point cloud acquisition device.

[0039] In this embodiment, environmental image data is detected based on a pre-trained smoke recognition neural network; if the smoke confidence level output by the neural network exceeds a preset threshold, it is determined that there is a medium that interferes with the detection capability of the first point cloud acquisition device.

[0040] It is understandable that, since the first point cloud acquisition device in this embodiment can be a lidar, its echo signal is easily attenuated or even rendered ineffective in dense smoke environments due to the scattering and absorption of laser light by smoke particles, thus significantly affecting the integrity and reliability of the first point cloud data. In contrast, the second point cloud acquisition device in this embodiment is generally a 4D millimeter-wave radar, which has a strong ability to penetrate interference media such as smoke and dust, and can provide effective obstacle detection information in scenarios where visible light and laser perception are limited. Therefore, in this embodiment, the processor introduces a deep learning-based smoke recognition model to perform real-time semantic analysis on environmental images acquired by the visible light camera, enabling it to quickly and accurately determine whether the current working environment is under smoke interference. Once the smoke confidence level exceeds a set threshold, it is determined that there is an interference medium affecting the normal operation of the lidar, triggering subsequent untrusted area division and multi-sensor trust weight switching mechanisms. This judgment logic not only improves the system's ability to perceive and adapt to complex fire environments but also provides a decision-making basis for distinguishing between the use of first and second point cloud data, thereby ensuring both perception accuracy and system robustness.

[0041] Preferably, in this embodiment of the application, the smoke recognition process can be deployed on the vehicle-mounted industrial control computer, combined with a lightweight neural network structure, to ensure that real-time inference and response can still be achieved under limited computing resources and communication constraints, thereby ensuring the vehicle's obstacle avoidance capability.

[0042] Step 140: Based on the spatial distribution information of the medium in the environmental image data, distinguish the untrusted point cloud region corresponding to the medium from the first point cloud data.

[0043] In this embodiment, when it is determined that there is a medium that interferes with the detection capability of the first point cloud acquisition device, the boundary of the smoke point cloud is determined based on the echo intensity in the first point cloud data; the spatial location and range of the smoke region are calculated based on the boundary of the smoke point cloud, and the minimum bounding box of the smoke region is determined; the minimum bounding box of the smoke region is taken as the untrusted point cloud region.

[0044] Understandably, because interfering media such as smoke significantly scatter and attenuate the detection signal of the first point cloud acquisition device (such as lidar), the returned point cloud data often exhibits abnormally low echo intensity or a sharp decrease in point cloud density, and may even contain a large number of false or missing points. Therefore, by analyzing the echo intensity distribution of each point in the first point cloud data, the area affected by smoke in the environment can be further confirmed.

[0045] Specifically, in this embodiment, after the processor identifies the presence of smoke based on environmental image data, the system maps the spatial distribution of smoke in the environmental image data to a point cloud space, and filters out a subset of point clouds suspected of being affected by smoke interference by combining the statistical characteristics of the point cloud echo intensity. Subsequently, the processor performs geometric modeling on this subset and calculates its minimum circumscribed bounding box (OBB), thereby accurately defining the spatial range of untrusted areas. This bounding box serves as the basis for subsequent sensor trust switching decisions, ensuring that a second point cloud acquisition device with stronger penetration is prioritized for obstacle perception within this area, thus guaranteeing the continuity and reliability of obstacle detection in complex fire scene environments.

[0046] Step 150: Within the spatial range corresponding to the untrusted point cloud region, obstacle detection is performed based on the second point cloud data first to obtain the first detection result.

[0047] In this embodiment, second point cloud data of the current moment and the previous historical moment are acquired; based on vehicle pose information, the second point cloud data of the historical moment are uniformly transformed to the coordinate system of the current moment to form a multi-frame fused point cloud; obstacle point cloud data of obstacles behind the smoke are extracted; the multi-frame fused point cloud is clustered based on a multi-frame joint clustering method to calculate the position, orientation and boundary information of the obstacles; combined with the changes in the vehicle's position and heading angle at the current moment and the previous historical moment, the obstacle point cloud data is projected in the vehicle coordinate system to generate a first detection result.

[0048] It is understandable that, in this embodiment, the second point cloud acquisition device has strong penetration capabilities, enabling it to effectively acquire spatial information of obstacles in complex fire environments such as smoke and dust. However, due to the low point cloud density, single-frame data may have perception blind spots or insufficient accuracy. Therefore, by fusing the second point cloud data from the current and historical moments, and combining it with high-precision vehicle pose information, the point cloud coverage density and obstacle contour integrity can be effectively improved. In this embodiment, the processor, based on a multi-frame joint clustering method, can use two frames of second point cloud data from the current and previous moments to identify static or dynamic obstacles behind the smoke, significantly enhancing the system's perception robustness in untrusted areas. Specifically, changes in vehicle position and heading angle, etc., can be determined based on the GPS (Global Positioning System).

[0049] Step 160: Within the spatial range excluding the untrusted point cloud region, obstacle detection is performed based on the first point cloud data first to obtain the second detection result.

[0050] In this embodiment of the application, the coordinates of the center point and boundary point of the obstacle are determined based on the first point cloud data; and the second detection result is determined based on the coordinates of the center point and boundary point.

[0051] Understandably, because the first point cloud acquisition device possesses high-resolution and high-precision spatial perception capabilities in smoke-free or low-interference environments, the point cloud data it generates can accurately depict the geometric contours and spatial distribution of obstacles. Therefore, in non-smoke or undisturbed areas, the system prioritizes using the first point cloud data for obstacle detection.

[0052] Specifically, in this embodiment, the processor calculates the OBB bounding box of the obstacle based on the first point cloud data, obtains the coordinates of the boundary points and center point of the OBB bounding box, and calculates the distance and orientation of the boundary points and center point from the vehicle, respectively, and finally forms a second detection result with high confidence, providing a basis for subsequent vehicle motion control.

[0053] Step 170: Generate the corresponding vehicle motion control command by combining the first and second detection results.

[0054] In this embodiment, the obstacle height is determined based on the first detection result and the second detection result. If the obstacle height is higher than a preset height threshold, the distance between the vehicle and the obstacle is determined. The vehicle's braking state and vehicle speed state are combined to generate a vehicle motion control command.

[0055] It is understood that, since the vehicle itself has a certain ground clearance, obstacles with a height lower than the preset threshold in this embodiment of the application do not require avoidance. When the obstacle height is higher than the preset height threshold, the processor continues to determine the distance between the vehicle and the obstacle based on the first and second detection results, and generates different control commands based on different distances and different vehicle motion states. Preferably, in this embodiment of the application, the preset height threshold can be 30 centimeters.

[0056] Figure 2 This diagram schematically illustrates the control logic of a remote-controlled driving safety control method according to an embodiment of this application. It is understood that in one embodiment, the processor first performs preliminary screening of detected obstacles, processing only those located in the non-driving direction and with a height less than 30 centimeters. Subsequently, the processor determines the distance to the nearest obstacle: if the distance is greater than 15 meters, the current cycle ends; if the distance is no greater than 15 meters but greater than 10 meters, a visual warning is issued on the display screen and a voice prompt is played. After issuing the warning, the processor determines whether the brake pedal is depressed: if it is depressed, the current cycle ends; if it is not depressed, a deceleration command is sent to reduce the vehicle speed. If the distance to the nearest obstacle is no greater than 10 meters, the processor determines whether the brake pedal is depressed: if it is depressed, the current cycle ends; if it is not depressed, the processor further determines whether the current vehicle speed is greater than 0: if the vehicle speed is greater than 0, a braking command is sent to stop the vehicle; if the vehicle speed is already 0, the braking command is canceled, and the vehicle remains stationary.

[0057] Please refer to the following: Figure 3 , Figure 3 The diagram illustrates a control architecture schematic of a remote-controlled driving safety control method according to an embodiment of this application. It can be noted that in one embodiment of this application, the processor first receives a clock signal from GNSS to unify the clock signals of the 4D millimeter-wave radar, lidar, and visible light camera. Then, the processor integrates the data from the three sensors via an industrial control computer and performs obstacle detection and obstacle avoidance strategy generation. Finally, the data is input to the PLC controller for vehicle motion control.

[0058] This application integrates multi-source sensing data from image acquisition devices, a first point cloud acquisition device, and a second point cloud acquisition device. Through spatiotemporal synchronization and a unified coordinate system, these three types of heterogeneous data are correlated to the same reference. Subsequently, based on the image data, the presence of media interfering with lidar detection is identified, and untrusted regions are separately delineated from the lidar point cloud. Within these regions, millimeter-wave radar data with high penetration is prioritized for obstacle detection, while in other regions, lidar data with higher accuracy is prioritized. Finally, the two types of detection results are fused to generate vehicle motion control commands. This application effectively overcomes the problem of single-sensor failure or reliability degradation in extreme environments such as dense smoke in fire scenes, significantly improving the accuracy of obstacle recognition and driving safety of remotely controlled fire trucks in complex scenarios.

[0059] Figure 4 A schematic block diagram of a remote-controlled driving safety control device 400 according to an embodiment of this application is shown. Figure 4 As shown, this application embodiment provides a remote driving safety control device 400, which may include: Memory 410 is configured to store instructions; The processor 420 is configured to retrieve instructions from the memory 410 and, when executing the instructions, to implement the aforementioned method for controlling the boom.

[0060] Figure 5 A schematic diagram illustrating the architecture of a remotely controlled driving vehicle 500 according to an embodiment of this application is shown. Figure 5 As shown in the illustration, this application also provides a remote-controlled driving vehicle 500, which may include: Remote driving safety control device 510 is used to execute remote driving safety control methods; Controller 520 is used to control vehicle speed.

[0061] This application also provides a machine-readable storage medium storing instructions that cause a machine to perform the aforementioned remote-controlled driving safety control method.

[0062] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0063] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0064] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0065] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0066] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0067] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0068] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0069] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0070] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A remote-controlled driving safety control method, characterized in that, include: Acquire environmental image data, first point cloud data, and second point cloud data collected by the image acquisition device, the first point cloud acquisition device, and the second point cloud acquisition device, respectively; Data from each acquisition device is synchronized in time and space and processed to unify the coordinate system, so as to associate data from different sources under the same time and space reference. Based on the environmental image data, identify whether there is a medium that interferes with the detection capability of the first point cloud acquisition device; Based on the spatial distribution information of the medium in the environmental image data, an untrusted point cloud region corresponding to the medium is distinguished from the first point cloud data; Within the spatial range corresponding to the untrusted point cloud region, obstacle detection is preferentially performed based on the second point cloud data to obtain a first detection result; Within the spatial range excluding the untrusted point cloud region, obstacle detection is preferentially performed based on the first point cloud data to obtain a second detection result; The corresponding vehicle motion control command is generated by combining the first detection result and the second detection result.

2. The safety control method according to claim 1, characterized in that, The process of performing spatiotemporal synchronization and coordinate system unification on data from various acquisition devices to correlate data from different sources to the same spatiotemporal reference includes: Based on an external clock source, the clock references of the image acquisition device, the first point cloud acquisition device, and the second point cloud acquisition device are synchronized. Based on the first point cloud data and the second point cloud data, multiple spatial feature points are selected to calculate feature vectors, and the spatial transformation matrix from the coordinate system of the second point cloud data to the coordinate system of the first point cloud data is determined. The first point cloud data is downsampled by pass-through filtering and voxel filtering to determine the processed first point cloud data. Based on the spatial transformation matrix, the second point cloud data and the processed first point cloud data are projected onto the coordinate system where the environmental image data is located.

3. The safety control method according to claim 2, characterized in that, The step of selecting multiple spatial feature points based on the first point cloud data and the second point cloud data to calculate feature vectors and determine the spatial transformation matrix from the coordinate system of the second point cloud data to the coordinate system of the first point cloud data includes: In the first point cloud data, at least three non-collinear spatial feature points are selected, a local coordinate system is constructed, and the corresponding three orthogonal feature vectors are calculated. In the second point cloud data, identify the same-name feature points that correspond to the spatial positions of feature points in the first point cloud data, and construct a local coordinate system based on the same-name feature points to calculate the corresponding three orthogonal feature vectors. Based on the correspondence between the two sets of feature vectors, the rotation matrix and translation vector that align the two coordinate systems are solved, thereby obtaining the spatial transformation matrix that transforms the second point cloud data coordinate system to the first point cloud data coordinate system.

4. The safety control method according to claim 1, characterized in that, The step of identifying whether there is a medium that interferes with the detection capability of the first point cloud acquisition device based on the environmental image data includes: The environmental image data is detected based on a pre-trained smoke recognition neural network; If the confidence level of the smoke output by the neural network exceeds a preset threshold, it is determined that there is a medium that interferes with the detection capability of the first point cloud acquisition device.

5. The safety control method according to claim 1, characterized in that, The step of distinguishing the untrusted point cloud region corresponding to the medium from the first point cloud data based on the spatial distribution information of the medium in the environmental image data includes: If it is determined that there is a medium that interferes with the detection capability of the first point cloud acquisition device, the boundary of the smoke point cloud is determined based on the echo intensity in the first point cloud data. Calculate the spatial location and extent of the smoke region based on the boundary of the smoke point cloud, and determine the minimum bounding box of the smoke region. The smallest bounding box of the smoke region is taken as the untrusted point cloud region.

6. The safety control method according to claim 1, characterized in that, Within the spatial range corresponding to the untrusted point cloud region, obstacle detection is preferentially performed based on the second point cloud data to obtain a first detection result, including: Obtain the second point cloud data at the current moment and the previous historical moment; Based on vehicle pose information, the second point cloud data at the historical moment is uniformly transformed to the coordinate system at the current moment to form a multi-frame fused point cloud; Extract obstacle point cloud data of obstacles behind the smoke; The multi-frame fused point cloud is clustered using a multi-frame joint clustering method to calculate the position, orientation, and boundary information of obstacles. The obstacle point cloud data is then projected onto the vehicle coordinate system, taking into account the changes in the vehicle's position and heading angle at the current and previous historical moments, to generate the first detection result.

7. The safety control method according to claim 1, characterized in that, The step of prioritizing obstacle detection based on the first point cloud data to obtain a second detection result within the spatial range excluding the untrusted point cloud region includes: The coordinates of the center point and boundary points of the obstacle are determined based on the first point cloud data; The second detection result is determined based on the coordinates of the center point and the boundary point.

8. The safety control method according to claim 1, characterized in that, The process of generating corresponding vehicle motion control commands by combining the first detection result and the second detection result includes: Based on the first and second detection results, the obstacle height is determined. If the obstacle height is higher than a preset height threshold, the distance between the vehicle and the obstacle is determined. Combined with the vehicle braking status and vehicle speed status, a vehicle motion control command is generated.

9. A remote-controlled driving safety control device, characterized in that, include: The memory is configured to store instructions; The processor is configured to retrieve the instructions from the memory and, when executing the instructions, to implement the remote driving safety control method according to any one of claims 1 to 8.

10. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores instructions for causing the machine to perform the remote driving safety control method according to any one of claims 1 to 8.

11. A remote-controlled vehicle, characterized in that, include: Remote driving safety control device, used to execute remote driving safety control methods; The controller is used to control the vehicle speed.

Images