Point cloud data processing methods, neural network training methods, and related equipment

Abstract

The embodiments of the present application disclose a method for processing point cloud data, a method for training a neural network, and related equipment. The method can be used in the field of processing point cloud data in the field of artificial intelligence. The method may include: obtaining point cloud data corresponding to the target environment, the point cloud data being divided into multiple target cubes; generating initial features of each target cube based on the initial information of the target points in each target cube; updating the initial features of the multiple target cubes according to the attention mechanism to obtain updated features of the multiple target cubes; performing feature processing operations on the updated features of the multiple target cubes to obtain prediction results corresponding to the point cloud data. This reduces the computer resources consumed in the entire point cloud data processing process, which is conducive to focusing on the important information carried by the point cloud data to improve the accuracy of the output prediction results.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a method for processing point cloud data, a method for training a neural network, and related equipment. Background Art

[0002] Artificial Intelligence (AI) is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that seeks to understand the essence of intelligence and develop new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0003] Using models to perform target detection on point cloud data corresponding to the vehicle's surroundings is a common processing method in the field of autonomous driving. Specifically, multiple target keypoints are sampled from the point cloud data. The features of multiple points whose distance from the target keypoint is less than the sampling radius are obtained. The features of the target keypoint are updated based on the features of multiple points whose distance from the target keypoint is less than the sampling radius, and the updated features of each target keypoint are obtained. Target detection is performed based on the updated features of each target keypoint to obtain a prediction result corresponding to the point cloud data.

[0004] However, since the entire point cloud needs to be sampled to determine the target key points, when the number of point clouds is large, the entire point cloud data processing process will consume a lot of computer resources. Summary of the Invention

[0005] The embodiments of the present application provide a method for processing point cloud data, a method for training a neural network, and related equipment, which perform target detection operations based on the features of multiple target cubes and generate updated features of the first cube according to the attention mechanism. This not only reduces the computer resources consumed during the entire point cloud data processing process, but also helps focus on the important information carried by the point cloud data to improve the accuracy of the prediction results.

[0006] To solve the above technical problems, the embodiments of the present application provide the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides a method for processing point cloud data, which can be used in the field of point cloud data processing in the field of artificial intelligence. The method may include: an execution device acquires point cloud data corresponding to a target environment, the point cloud data including initial information of multiple target points; inputs the point cloud data to be processed into a target model, so as to divide the point cloud data to be processed into multiple target cubes through the target model, wherein there are S target points in each target cube, and S is an integer greater than or equal to 0. The execution device generates initial features of each target cube through the target model based on the initial information of the target points in each target cube, and updates the initial features of the multiple target cubes according to the attention mechanism to obtain updated features of the multiple target cubes. The execution device performs feature processing operations through the target model based on the updated features of the multiple target cubes to obtain a prediction result corresponding to the point cloud data, and the prediction result represents information of at least one object in the target environment; further, the prediction result is used to indicate the position information or category information of at least one object in the target environment.

[0008] In this implementation, after generating the initial features of the first cube through the target model based on the initial information of the target point in the first cube, the updated features of the first cube are generated according to the attention mechanism based on the initial features of the first cube, and then target detection is performed based on the updated features of multiple target cubes, which is conducive to focusing on the important information carried by the point cloud data to improve the accuracy of the prediction results.

[0009] In a possible implementation of the first aspect, a target model includes P first feature update modules, where P is an integer greater than or equal to 1. An execution device updates the initial features of multiple target cubes according to an attention mechanism, including: when the first cube satisfies a first preset condition, the execution device updates the features of each first cube in the initial features of the multiple target cubes through the first feature update module according to the attention mechanism. The first cube is any one of the multiple target cubes, the first preset condition is that the first cube is a non-empty cube, that is, the first preset condition is that the first cube is not an empty cube. Furthermore, a non-empty cube has the opposite meaning to an empty cube, and the features of each empty cube are preset information, that is, the features of each non-empty cube are not preset information. Furthermore, the initial features of the first cube and the updated features of the first cube can be specifically represented as vectors of length d, and the preset information can also be represented as a vector of length d. As an example, the preset information can be represented as a vector including d zeros, or as a vector including d ones, or as a vector including d different numbers, etc.

[0010] In the embodiment of the present application, the feature update of the first cube will be performed according to the attention mechanism only when the first cube is not an empty cube. That is, after the multiple target cubes corresponding to the point cloud data to be processed are updated by the first feature update module, the distribution of hollow cubes and non-empty cubes in the multiple target cubes corresponding to the point cloud data to be processed will not change, thereby realizing the feature update of multiple target cubes and helping to retain the edge features of the object.

[0011] In one possible implementation of the first aspect, the method is applied to an execution device, which utilizes a hash table to record empty cubes among multiple target cubes. The hash table also records non-empty cubes among the multiple target cubes. Furthermore, the hash table corresponding to the multiple target cubes may include two components: a key and a value. The key component in the hash table may be obtained based on index information of the target cubes. If a target cube is an empty cube, the value corresponding to the empty target cube may be a preset value. If a target cube is not an empty cube, the value corresponding to the non-empty cube may be the order of the target cube among all non-empty cubes.

[0012] In an embodiment of the present application, the method of using a hash table to record empty cubes in multiple target cubes can greatly improve the efficiency of the process of obtaining features of non-empty cubes, thereby helping to improve the efficiency of the process of target detection based on point cloud data.

[0013] In a possible implementation of the first aspect, the target model includes Q second feature update modules, where Q is an integer greater than or equal to 1. The execution device updates the initial features of each of the initial features of multiple target cubes according to an attention mechanism, including: when the first cube meets a second preset condition, the execution device updates the features of each first cube in the initial features of the multiple target cubes using the second feature update module according to the attention mechanism. The second preset condition is that there is a non-empty cube in a cube set, the cube set including the first cube and at least one second cube corresponding to the first cube, and the at least one second cube is determined from the multiple target cubes according to the attention mechanism.

[0014] In the embodiment of the present application, only when the first cube and multiple second cubes are empty cubes will the features of the first cube not be updated through the second feature updating module, which is conducive to updating the features of more target cubes and improving the feature acquisition capability of the target model.

[0015] In a possible implementation of the first aspect, the target model includes P first feature update modules and Q second feature update modules, and the execution device updates the initial features of each target cube in the initial features of multiple target cubes according to the attention mechanism, including: when the first cube meets the first preset condition, the execution device updates the features of each first cube in the initial features of the multiple target cubes through the first feature update module according to the attention mechanism. The first preset condition is that the first cube is a non-empty cube, and the features of each non-empty cube are not preset information. When the first cube meets the second preset condition, the execution device updates the features of each first cube in the initial features of the multiple target cubes through the second feature update module according to the attention mechanism. The second preset condition is that there is a non-empty cube in the cube set, and the cube set includes a first cube and at least one second cube corresponding to the first cube, and the at least one second cube is determined from the multiple target cubes according to the attention mechanism.

[0016] In a possible implementation of the first aspect, the value of P is greater than the value of Q. In an embodiment of the present application, the target model may include P first feature update modules and Q second feature update modules, and the value of P is greater than the value of Q. This improves the feature acquisition capability of the target model while preserving the edge features of the object as much as possible, which is conducive to improving the accuracy of the final prediction result.

[0017] In a possible implementation of the first aspect, the execution device updates the features of each first cube in the initial features of multiple target cubes according to the attention mechanism, which may include: the execution device obtains at least one second cube corresponding to the first cube through the first local attention operator, the at least one second cube includes all target cubes within a preset range around the first cube, and the distance between each second cube and the first cube within the preset range around the first cube is less than or equal to a preset distance threshold, and the value of the preset distance threshold can be 1 target cube, 2 target cubes, 3 target cubes or other values; based on the features of the at least one second cube, the first cube is updated through the first local attention operator. And / or,

[0018] The execution device obtains at least one second cube corresponding to the first cube through the second local attention operator, where the at least one second cube is obtained by sampling multiple target cubes with the first cube as the sampling center; and updates the features of the first cube through the second local attention operator according to the features of the at least one second cube.

[0019] In an embodiment of the present application, two specific positions of at least one second cube corresponding to the first cube are provided, which improves the implementation flexibility of the present solution; and at least one second cube is obtained by sampling multiple target cubes included in the data to be processed, that is, the second cube may not be limited to the cube around the first cube, which greatly expands the receptive field of the target model, thereby improving the feature acquisition capability of the target model, which is conducive to improving the accuracy of the final prediction result.

[0020] In a possible implementation of the first aspect, the execution device updates the features of each first cube in multiple target cubes according to the attention mechanism, and also includes: the execution device generates features of multiple third cubes based on the features of the second cube, and a third cube includes multiple second cubes; according to the features of each third cube, the feature of each first cube in the multiple target cubes is updated based on the global attention mechanism, and the global attention mechanism is one of the attention mechanisms.

[0021] In an embodiment of the present application, the multiple second cubes corresponding to the first cube include all target cubes, that is, the features of all target cubes corresponding to the point cloud data to be processed are used to update the features of the first cube, further expanding the receptive field of the target model and further improving the feature acquisition capability of the target model; in addition, a third cube includes multiple second cubes, that is, all target cubes are divided into multiple third cubes, and then the features of the first cube are updated using the features of the multiple third cubes, which is beneficial to improving the efficiency of the feature update process.

[0022] In a possible implementation of the first aspect, the characteristics of the third cube may be statistical values ​​of characteristics of multiple second cubes included in the third cube, and the aforementioned statistical values ​​may specifically include any one or more of the following statistical characteristics: maximum value, minimum value, mean, variance or other types of statistical characteristics, etc.; the coordinates of the third cube may be the coordinates of a second cube among the multiple second cubes included in the third cube, and the specific coordinates of the second cube selected as the coordinates of the third cube can be determined according to the type of statistical characteristics adopted by the characteristics of the third cube.

[0023] In a possible implementation of the second aspect, the execution device performs a feature processing operation on the updated features of the multiple target cubes to obtain a prediction result corresponding to the point cloud data, including:

[0024] The execution device performs a target detection operation on the updated features of the multiple target cubes to obtain a prediction result corresponding to the point cloud data. The prediction result is used to indicate the position information of at least one object in the target environment. The execution device configured with the target model can be any of the following devices: a vehicle, a mobile robot, or a monitoring data processing device; the prediction result is used to indicate any of the following information: the position of at least one object in the vehicle's surrounding environment, the position of at least one object in the mobile robot's surrounding environment, or the environment of at least one object in the monitoring environment. Or,

[0025] The execution device performs a face recognition operation on the updated features of the multiple target cubes to obtain a prediction result corresponding to the point cloud data. The information of at least one object in the target environment includes category information of at least one object in the target environment. The category information of at least one object in the target environment is used to indicate whether the face match is successful or failed. The execution device configured with the target model can be any of the following devices: a terminal device, a smart door lock, a monitoring data processing device; or

[0026] The execution device performs a posture recognition operation based on the updated features of the multiple target cubes to obtain a prediction result corresponding to the point cloud data. The information of at least one object in the target environment includes the physical features of at least one human body in the target environment and / or the gait features of at least one human body in the target environment when walking. Furthermore, the physical features may refer to visually or tactilely perceptible appearance features of the target object. The physical features may be inherent characteristics of the target object, such as fatness, height, muscularity, etc. Gait features refer to information related to walking speed and walking posture.

[0027] In the embodiments of the present application, multiple specific application scenarios of this solution are provided, which improves the implementation flexibility of this solution.

[0028] In a second aspect, an embodiment of the present application provides a method for processing point cloud data, which can be used in the field of point cloud data processing in the field of artificial intelligence. The method may include: an execution device acquires point cloud data corresponding to the surrounding environment of a target vehicle, wherein the point cloud data includes initial information of multiple target points, and the point cloud data is divided into multiple target cubes, and there are S target points in one target cube, where S is an integer greater than or equal to 0; the execution device generates initial features of each target cube based on the initial information of the target points in each target cube; the initial features of the multiple target cubes are updated according to the attention mechanism to obtain updated features of the multiple target cubes; the execution device performs a target detection operation on the updated features of the multiple target cubes to obtain a prediction result corresponding to the point cloud data, and the prediction result is used to indicate the position of at least one object in the surrounding environment of the target vehicle.

[0029] In a possible implementation of the second aspect, the execution device updates the initial features of multiple target cubes according to the attention mechanism, including: when the first cube meets the first preset condition, updating the initial features of each first cube in the multiple target cubes according to the attention mechanism, the first cube is any one of the multiple target cubes, the first preset condition is that the first cube is a non-empty cube, and the feature of the non-empty cube is not preset information.

[0030] In a possible implementation of the second aspect, the execution device updates the initial features of multiple target cubes according to the attention mechanism, and also includes: when the first cube meets the second preset condition, updating the features of each first cube in the multiple target cubes according to the attention mechanism; wherein the second preset condition is that there is a non-empty cube in the cube set, the cube set includes the first cube and at least one second cube corresponding to the first cube, and the at least one second cube is determined from the multiple target cubes according to the attention mechanism.

[0031] The execution device provided in the second aspect of the embodiment of the present application can also execute the steps executed by the execution device in each possible implementation method of the first aspect. For the specific implementation steps of the second aspect of the embodiment of the present application and the various possible implementation methods of the second aspect, as well as the beneficial effects brought about by each possible implementation method, you can refer to the description of the various possible implementation methods in the first aspect, and will not repeat them one by one here.

[0032] In a third aspect, an embodiment of the present application provides a neural network training method that can be used in the field of processing point cloud data in the field of artificial intelligence. The method may include: a training device acquires point cloud data corresponding to a target environment, wherein the point cloud data includes initial information of multiple target points, and the point cloud data is divided into multiple target cubes, and there are S target points in one target cube, where S is an integer greater than or equal to 0; the training device generates initial features of each target cube through a to-be-trained model based on the initial information of the target points in each target cube, and updates the initial features of the multiple target cubes according to an attention mechanism to obtain updated features of the multiple target cubes; based on the updated features of the multiple target cubes, a feature processing operation is performed through the to-be-trained model to obtain a prediction result corresponding to the point cloud data; the training device trains the to-be-trained model according to a target loss function, wherein the target loss function is used to indicate the similarity between the prediction result and the expected result corresponding to the point cloud data.

[0033] In a possible implementation of the third aspect, the training device updates the initial features of multiple target cubes according to the attention mechanism, including: when the first cube meets the first preset condition, updating the initial features of each first cube in the multiple target cubes according to the attention mechanism, the first cube is any one of the multiple target cubes, the first preset condition is that the first cube is a non-empty cube, and the feature of the non-empty cube is not preset information.

[0034] In a possible implementation of the third aspect, the training device updates the initial features of multiple target cubes according to the attention mechanism, and also includes: when the first cube meets the second preset condition, updating the features of each first cube in the multiple target cubes according to the attention mechanism; wherein the second preset condition is that there is a non-empty cube in the cube set, the cube set includes the first cube and at least one second cube corresponding to the first cube, and the at least one second cube is determined from the multiple target cubes according to the attention mechanism.

[0035] The training device provided in the third aspect of the embodiment of the present application can also execute the steps performed by the execution device in each possible implementation method of the first aspect. For the specific implementation steps of the third aspect of the embodiment of the present application and the various possible implementation methods of the third aspect, as well as the beneficial effects brought about by each possible implementation method, you can refer to the description of the various possible implementation methods in the first aspect, and will not be repeated here one by one.

[0036] In a fourth aspect, an embodiment of the present application provides a method for processing point cloud data, which can be used in the field of point cloud data processing in the field of artificial intelligence. The method may include: an execution device detecting an operation indicating face recognition; the execution device outputting a face recognition result in response to the operation indicating face recognition; wherein the face recognition result includes a successful match and a failed match, and the face recognition result is obtained based on the updated features of multiple target cubes, the updated features of the target cube are obtained by updating the initial features of the target cube according to the attention mechanism, the initial features of the target cube are obtained based on the collected point cloud data corresponding to the target environment, the point cloud data includes initial information of multiple target points, the point cloud data is divided into multiple target cubes, there are S target points in one target cube, and S is an integer greater than or equal to 0.

[0037] In one possible implementation of the fourth aspect, detecting an operation indicating facial recognition includes detecting any of the following operations: unlocking a mobile phone, making a payment, or opening encrypted data; or detecting a door opening operation input by a user; or detecting a face matching operation input by a user. In the embodiments of the present application, multiple application scenarios for detecting facial recognition operations are provided, increasing the implementation flexibility of the present solution.

[0038] The execution device provided in the fourth aspect of the embodiment of the present application can also execute the steps performed by the execution device in each possible implementation method of the second aspect. For the specific implementation steps of the fourth aspect of the embodiment of the present application and the various possible implementation methods of the fourth aspect, as well as the beneficial effects brought about by each possible implementation method, you can refer to the description of the various possible implementation methods in the second aspect, and will not be repeated here one by one.

[0039] In a fifth aspect, an embodiment of the present application provides a point cloud data processing device that can be used in the field of point cloud data processing in the field of artificial intelligence. The point cloud data processing device includes: an acquisition module for acquiring point cloud data corresponding to a target environment, wherein the point cloud data includes initial information of multiple target points, and the point cloud data is divided into multiple target cubes, each target cube has S target points, and S is an integer greater than or equal to 0; a generation module for generating initial features of each target cube based on the initial information of the target points in each target cube; an update module for updating the initial features of multiple target cubes according to the attention mechanism to obtain updated features of multiple target cubes; a processing module for performing feature processing operations on the updated features of multiple target cubes to obtain prediction results corresponding to the point cloud data, and the prediction results represent information of at least one object in the target environment.

[0040] The point cloud data processing device provided in the fifth aspect of the embodiment of the present application can also execute the steps performed by the execution device in each possible implementation method of the first aspect. For the specific implementation steps of the fifth aspect of the embodiment of the present application and the various possible implementation methods of the fifth aspect, as well as the beneficial effects brought about by each possible implementation method, you can refer to the description of the various possible implementation methods in the first aspect, and will not repeat them one by one here.

[0041] In a sixth aspect, an embodiment of the present application provides a neural network training device that can be used in the field of processing point cloud data in the field of artificial intelligence. The neural network training device may include: an acquisition module for acquiring point cloud data corresponding to a target environment, wherein the point cloud data includes initial information of multiple target points, and the point cloud data is divided into multiple target cubes, and there are S target points in one target cube, where S is an integer greater than or equal to 0; a generation module for generating initial features of each target cube through a to-be-trained model based on the initial information of the target points in each target cube; an update module for updating the initial features of multiple target cubes according to an attention mechanism to obtain updated features of multiple target cubes; a processing module for performing feature processing operations on the to-be-trained model based on the updated features of multiple target cubes to obtain prediction results corresponding to the point cloud data; a training module for training the to-be-trained model according to a target loss function, wherein the target loss function is used to indicate the similarity between the prediction results and the expected results corresponding to the point cloud data.

[0042] The neural network training device provided in the sixth aspect of the embodiment of the present application can also execute the steps executed by the training device in each possible implementation method of the third aspect. For the specific implementation steps of the sixth aspect of the embodiment of the present application and the various possible implementation methods of the sixth aspect, as well as the beneficial effects brought about by each possible implementation method, reference can be made to the description of the various possible implementation methods in the third aspect, and no further details will be given here.

[0043] In a seventh aspect, an embodiment of the present application provides a point cloud data processing device that can be used in the field of point cloud data processing in the field of artificial intelligence. The point cloud data processing device includes: a detection module for detecting an operation indicating face recognition; an output module for outputting a face recognition result in response to the operation indicating face recognition; wherein the face recognition result includes a successful match and a failed match, and the face recognition result is obtained based on the updated features of multiple target cubes, the updated features of the target cube are obtained by updating the initial features of the target cube according to the attention mechanism, and the initial features of the target cube are obtained based on the collected point cloud data corresponding to the target environment, the point cloud data includes initial information of multiple target points, and the point cloud data is divided into multiple target cubes. There are S target points in one target cube, and S is an integer greater than or equal to 0.

[0044] The point cloud data processing device provided in the seventh aspect of the embodiment of the present application can also execute the steps performed by the execution device in each possible implementation method of the fourth aspect. For the specific implementation steps of the seventh aspect of the embodiment of the present application and the various possible implementation methods of the seventh aspect, as well as the beneficial effects brought about by each possible implementation method, you can refer to the description of the various possible implementation methods in the third aspect, and will not go into details here.

[0045] In an eighth aspect, an embodiment of the present application provides a computer program product, which, when running on a computer, enables the computer to execute the point cloud data processing method described in the first, second or fourth aspect above, or enables the computer to execute the neural network training method described in the third aspect above.

[0046] In the ninth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the program is run on a computer, the computer executes the point cloud data processing method described in the first, second or fourth aspect, or the neural network training method described in the third aspect.

[0047] In the tenth aspect, an embodiment of the present application provides an execution device, which may include a processor, the processor and a memory are coupled, and the memory stores program instructions. When the program instructions stored in the memory are executed by the processor, the point cloud data processing method described in the first aspect, second aspect or fourth aspect above is implemented.

[0048] In the eleventh aspect, an embodiment of the present application provides a training device, which may include a processor, the processor and a memory are coupled, and the memory stores program instructions. When the program instructions stored in the memory are executed by the processor, the neural network training method described in the third aspect above is implemented.

[0049] In the twelfth aspect, an embodiment of the present application provides a circuit system, which includes a processing circuit, and the processing circuit is configured to execute the point cloud data processing method described in the first aspect, the second aspect or the fourth aspect above, or the processing circuit is configured to execute the neural network training method described in the third aspect above.

[0050] In a thirteenth aspect, an embodiment of the present application provides a chip system, which includes a processor for implementing the functions involved in the above-mentioned various aspects, for example, sending or processing the data and / or information involved in the above-mentioned method. In one possible design, the chip system also includes a memory, which is used to store program instructions and data necessary for the server or communication device. The chip system can be composed of a chip, or it can include a chip and other discrete devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1a A schematic diagram of the structure of the artificial intelligence main framework provided in the embodiment of the present application;

[0052] Figure 1b A diagram of an application scenario of the point cloud data processing method provided in an embodiment of the present application;

[0053] Figure 2a A system architecture diagram of a point cloud data processing system provided in an embodiment of the present application;

[0054] Figure 2b A schematic diagram of a process for processing point cloud data provided in an embodiment of the present application;

[0055] Figure 2c A schematic diagram of a target model in the point cloud data processing method provided in an embodiment of the present application;

[0056] Figure 3 Another flowchart of the method for processing point cloud data provided in an embodiment of the present application;

[0057] Figure 4 A schematic diagram of the point cloud data to be processed and the target cube in the point cloud data processing method provided in an embodiment of the present application;

[0058] Figure 5 A schematic diagram of multiple second cubes corresponding to the first local attention operator in the method for processing point cloud data to be processed provided by an embodiment of the present application;

[0059] Figure 6 A schematic diagram of multiple second cubes corresponding to the second local attention operator in the method for processing point cloud data to be processed provided by an embodiment of the present application;

[0060] Figure 7 A schematic diagram of multiple second cubes corresponding to the global attention operator in the method for processing point cloud data to be processed provided in an embodiment of the present application;

[0061] Figure 8 A schematic diagram of multiple second cubes corresponding to the first local attention operator and the second local attention operator in the method for processing point cloud data to be processed provided by an embodiment of the present application;

[0062] Figure 9a A schematic diagram of a first feature updating module in the method for processing point cloud data to be processed provided in an embodiment of the present application;

[0063] Figure 9b A schematic diagram of a first feature updating module in the method for processing point cloud data to be processed provided in an embodiment of the present application;

[0064] Figure 10 A schematic diagram of using a hash table to determine whether multiple target cubes are empty cubes in the method for processing point cloud data to be processed provided in an embodiment of the present application;

[0065] Figure 11 A schematic diagram of a feature updating module of a target model in a method for processing point cloud data to be processed provided in an embodiment of the present application;

[0066] Figure 12 A schematic diagram of another flow chart of a method for processing point cloud data provided in an embodiment of the present application;

[0067] Figure 13 A flowchart of a neural network training method provided in an embodiment of the present application;

[0068] Figure 14 Another flowchart of the method for processing point cloud data provided in an embodiment of the present application;

[0069] Figure 15 A schematic diagram of an operation indicating face recognition detected in the point cloud data processing method provided in an embodiment of the present application;

[0070] Figure 16 A schematic diagram of the structure of a point cloud data processing device provided in an embodiment of the present application;

[0071] Figure 17 Another structural diagram of the point cloud data processing device provided in an embodiment of the present application;

[0072] Figure 18 A schematic diagram of the structure of a neural network training device provided in an embodiment of the present application;

[0073] Figure 19 Another structural diagram of the point cloud data processing device provided in an embodiment of the present application;

[0074] Figure 20 A schematic diagram of the structure of an execution device provided in an embodiment of the present application;

[0075] Figure 21 A schematic diagram of the structure of the training device provided in an embodiment of the present application;

[0076] Figure 22 A schematic diagram of the structure of the chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0077] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0078] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0079] First, the overall workflow of the artificial intelligence system is described. Figure 1a , Figure 1a The following diagram illustrates a structural diagram of the AI ​​framework. This framework is explained below from two perspectives: the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis). The "intelligent information chain" reflects the entire process from data acquisition to processing. For example, it encompasses the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. Throughout this process, data undergoes a condensed progression from "data-information-knowledge-wisdom." The "IT value chain," encompassing the entire process from the underlying infrastructure of human intelligence, information (provided and processed by technology), to the system's industrial ecosystem, reflects the value that AI brings to the information technology industry.

[0080] (1) Infrastructure

[0081] The infrastructure provides computing power support for artificial intelligence systems, enabling communication with the outside world and providing support through the basic platform. Communication with the outside world is achieved through sensors; computing power is provided by intelligent chips, which can specifically adopt hardware acceleration chips such as central processing units (CPUs), embedded neural network processing units (NPUs), graphics processing units (GPUs), application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs); the basic platform includes related platform guarantees and support such as distributed computing frameworks and networks, and can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to obtain data, and this data is provided to the intelligent chips in the distributed computing system provided by the basic platform for calculation.

[0082] (2) Data

[0083] Data above the infrastructure layer represents data sources for AI. This data includes graphics, images, voice, and text, as well as IoT data from traditional devices. This includes business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0084] (3) Data processing

[0085] Data processing generally includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.

[0086] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0087] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0088] Decision-making refers to the process of making decisions after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.

[0089] (4) General ability

[0090] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0091] (5) Smart products and industry applications

[0092] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart terminals, smart manufacturing, smart transportation, smart homes, smart medical care, smart security, autonomous driving, smart cities, etc.

[0093] The present application can be applied to the field of processing point cloud data in the field of artificial intelligence. Specifically, it can be applied to target detection, face recognition, posture recognition or other application scenarios using point cloud data. The specific application scenarios of the embodiments of the present application are illustrated below.

[0094] Application Scenario 1: Object Detection

[0095] In the embodiment of the present application, as an example, in the field of autonomous driving, the autonomous driving vehicle collects point cloud data corresponding to the environment around the vehicle through sensors, and performs target detection based on the collected point cloud data through the target model to obtain a prediction result corresponding to the point cloud data. The prediction result is used to indicate the position of at least one object in the environment around the target vehicle. The autonomous driving vehicle can plan its own driving path based on the aforementioned prediction result. For a more intuitive understanding of this solution, please refer to Figure 1b , Figure 1b An application scenario diagram of the point cloud data processing method provided in an embodiment of the present application. As shown in the figure, the vehicle in the left figure can collect the point cloud data shown in the right figure through the sensor, and the vehicle performs target detection through the target model based on the collected point cloud data. It should be understood that the example here is only for the convenience of understanding this solution and is not used to limit this solution.

[0096] It should be noted that the above-mentioned vehicles can be cars, trucks, motorcycles, buses, ships, airplanes, helicopters, lawn mowers, recreational vehicles, amusement park vehicles, construction equipment, trams, golf carts, trains, and carts, etc., and the embodiments of the present application do not make special limitations.

[0097] As another example, in the field of smart homes, a mobile robot (such as a sweeping robot, a tutoring robot, or other movable robots, etc.) can collect point cloud data corresponding to the robot's surrounding environment, and perform target detection through a target model based on the collected point cloud data to obtain a prediction result corresponding to the point cloud data. The prediction result is used to indicate the position of at least one obstacle around the mobile robot.

[0098] As another example, in the field of intelligent manufacturing, monitoring equipment configured in a factory can collect point cloud data corresponding to the monitoring area, and perform target detection through a target model based on the collected point cloud data to obtain a prediction result corresponding to the point cloud data. The prediction result is used to indicate the position of at least one object in the monitoring area.

[0099] Application scenario 2: face recognition

[0100] In an embodiment of the present application, as an example, in the field of smart terminals, when the terminal device needs to unlock the screen, make a payment, open encrypted data, or perform other operations, the terminal device can collect point cloud data corresponding to the surrounding environment, and perform identity authentication through face recognition based on the collected point cloud data, so that only specific users can successfully perform the aforementioned operations.

[0101] As another example, in the field of smart security, a smart door lock can collect point cloud data corresponding to the surrounding environment, and perform identity authentication through facial recognition based on the collected point cloud data, so that only specific users can successfully open the smart door lock.

[0102] As another example, in the field of smart security, intelligent monitoring can collect point cloud data corresponding to the surrounding environment. When there is a target face in the surrounding environment, the feature information of the target face can be obtained based on the collected point cloud data, and compared with the facial features of multiple people in the database to obtain the identity information of the target face.

[0103] Application Scenario 3: Gesture Recognition

[0104] In the embodiments of the present application, in smart homes, smart terminals or other fields, posture recognition can be performed based on the collected point cloud data of the human body to obtain posture features corresponding to the human body.

[0105] Among them, body characteristics may include physical characteristics and / or gait characteristics when walking. Furthermore, physical characteristics may refer to the appearance characteristics of the target object that can be perceived visually and tactilely. Physical characteristics may be inherent characteristics of the target object, such as fatness, height, muscle strength, etc. Gait characteristics refer to information related to walking speed and walking posture. Since different target objects have subtle differences in muscle strength, tendon and bone length, bone density, visual acuity, coordination ability, weight, center of gravity, degree of muscle or bone damage, physiological conditions and personal walking style, the gait characteristics of different target objects are different.

[0106] As an example, in a smart home scenario, the movement of the target object may be the movement when walking on the ground, and the body characteristics may include the shape characteristics when walking and the gait characteristics when walking. In a smart cockpit scenario, the movement of the target object may be the movement when getting on and off the car, and the body characteristics may include the shape characteristics when getting on and off the car, and so on.

[0107] It should be noted that the above examples are only for the purpose of facilitating the understanding of the application scenarios of this solution and are not intended to limit this solution. All application scenarios of the embodiments of this application are not listed here. In all of the above scenarios, the entire point cloud data processing process consumes a large amount of computer resources.

[0108] In order to solve the above problems, before introducing the method for processing point cloud data provided by the embodiment of the present application in detail, Figure 2a The point cloud data processing system provided in the embodiment of this application is introduced. Figure 2a , Figure 2a A system architecture diagram of a point cloud data processing system provided in an embodiment of the present application, Figure 2a In the figure, the point cloud data processing system 200 includes an execution device 210, a training device 220, a database 230 and a data storage system 240, and the execution device 210 includes a computing module 211.

[0109] The database 230 stores a training data set, and the training device 220 generates a target model / rule 201 and iteratively trains the target model / rule 201 using the training data set in the database 230 to obtain a mature target model / rule 201. Furthermore, the target model / rule 201 can be implemented using a neural network or a non-neural network model.

[0110] The mature target model / rule 201 obtained by the training device 220 can be applied to various systems or devices, such as vehicles, mobile robots, monitoring data processing equipment, or other devices that process point cloud data. The execution device 210 can access data, code, etc. from the data storage system 240, or store data, instructions, etc. in the data storage system 240. The data storage system 240 can be located within the execution device 210, or it can be an external memory device relative to the execution device 210.

[0111] The execution device 210 can obtain point cloud data corresponding to the target environment, and the calculation module 211 can perform target detection based on the collected point cloud data through the target model / rule 201 to obtain a prediction result corresponding to the point cloud data. The prediction result is used to indicate the position of at least one object in the surrounding environment of the target vehicle.

[0112] The execution device 210 can execute the following steps through the target model / rule 201. For details, please refer to Figure 2b , Figure 2b A flow chart of a method for processing point cloud data provided in an embodiment of the present application, A1. The execution device 210 obtains point cloud data corresponding to the target environment, wherein the point cloud data includes initial information of multiple target points, and the point cloud data is divided into multiple target cubes, each target cube has S target points, and S is an integer greater than or equal to 0; A2. The execution device 210 generates initial features of each target cube based on the initial information of the target points in each target cube; A3. The execution device 210 updates the initial features of the multiple target cubes according to the attention mechanism to obtain updated features of the multiple target cubes; A4. The execution device 210 performs feature processing operations on the updated features of the multiple target cubes to obtain prediction results corresponding to the point cloud data, and the prediction results represent information of at least one object in the target environment.

[0113] Further, to facilitate understanding of this solution, please refer to Figure 2c , Figure 2c A schematic diagram of a target model in a method for processing point cloud data provided in an embodiment of the present application, as shown in the figure, after acquiring point cloud data corresponding to the surrounding environment of a target vehicle, the execution device divides the point cloud data into multiple target cubes through the target model, and generates initial features of multiple target cubes. The execution device updates the features of the target cubes through the target model to obtain updated features of multiple target cubes. The execution device performs a target detection operation based on the updated features of the multiple target cubes to obtain a prediction result corresponding to the point cloud data. The prediction result is used to indicate the position of at least one object in the surrounding environment of the target vehicle. It should be understood that Figure 2cThe examples are only for facilitating understanding of this solution and are not intended to limit this solution.

[0114] In an embodiment of the present application, the point cloud data corresponding to the surrounding environment of the target vehicle is divided into multiple cubes, and then the target detection operation is performed based on the features of the multiple target cubes. That is, the minimum granularity of the processing object in the embodiment of the present application is no longer a target point, but a cube, which reduces the computer resources consumed in the entire point cloud data processing process and is conducive to improving the operating efficiency of the target model; and according to the initial features of the first cube, the updated features of the first cube are generated according to the attention mechanism, and then the target detection is performed according to the updated features of the multiple target cubes, which is conducive to focusing on the important information carried by the point cloud data and improving the accuracy of the prediction results.

[0115] In some embodiments of this application, please refer to Figure 2a The "user" can interact directly with the execution device 210. It is worth noting that Figure 2a This is merely a schematic diagram of the architecture of a point cloud data processing system provided by an embodiment of the present invention. The positional relationships between the devices, components, modules, etc. shown in the diagram do not constitute any limitation. For example, in other embodiments of the present application, the execution device 210 and the client device may be separate devices, with the execution device 210 being configured with an input / output (I / O) interface, through which the execution device 210 exchanges data with the client device.

[0116] To facilitate understanding of the present solution, before providing a detailed introduction to the method for processing point cloud data provided in the embodiments of the present application, the following first introduces relevant terms and concepts that may be involved in the embodiments of the present application.

[0117] (1) Attention Mechanism

[0118] In deep learning, the attention mechanism can be broadly defined as a weight vector that describes importance: in order to predict or infer an element, this weight vector is used. For example, for a pixel in an image or a word in a sentence, the attention vector can be used to quantitatively estimate the correlation between the target element and other elements, and the weighted sum of the attention vectors is used as an approximation of the target.

[0119] The attention mechanism in deep learning simulates the attention mechanism of the human brain. For example, when a person looks at a painting, although their eyes can see the entire painting, when they look closely, their eyes actually focus on only a small part of the pattern. At this time, the human brain focuses primarily on this small part. In other words, when a person observes an image carefully, the human brain's attention is not evenly distributed across the entire image; instead, it is weighted differently. This is the core idea of ​​the attention mechanism.

[0120] Simply put, the human visual processing system tends to selectively focus on certain parts of an image while ignoring other irrelevant information, thereby facilitating human perception. Similarly, in deep learning's attention mechanism, in some problems involving language, speech, or vision, certain parts of the input may be more relevant than others. Therefore, through the attention mechanism in the attention model, the attention model can dynamically focus on only the parts of the input that help effectively perform the task at hand.

[0121] (2) Feature Update

[0122] The process of generating initial features for the first cube using the target model and the process of updating the features of the first cube to obtain updated features of the first cube are both included in the process of extracting features from point cloud data using the target model. Feature extraction refers to the process of converting raw point cloud data into feature data suitable for modeling while still accurately describing the original dataset.

[0123] (3) Empty cube

[0124] In the process of generating the initial features of the first cube through the target model, the empty cube refers to a cube containing 0 points, and the execution device will determine the initial features of the empty cube as preset information; in the process of updating the features of multiple target cubes, the features of the empty cube are preset information, that is, the empty cube contains 0 points and is skipped during the feature update process.

[0125] (4) Non-empty cube

[0126] "Non-empty cube" has the opposite meaning to "empty cube". "Non-empty cube" means it is not an empty cube.

[0127] (5) Loss function

[0128] During neural network training, to ensure the neural network's output is as close as possible to the desired predicted value, the weight vectors of each layer are updated based on the difference between the predicted value and the desired expected value. (Of course, before the first update, there is usually an initialization process, which pre-configures the parameters for each layer in the deep neural network.) For example, if the network's predicted value is too high, the weight vectors are adjusted to predict a lower value. This adjustment is repeated until the deep neural network can predict the desired expected value or a value very close to the desired expected value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value." This is the loss function or objective function, which is an important equation used to measure the difference between the predicted value and the expected value. For example, the loss function output value (loss) indicates a greater difference, and neural network training becomes a process of minimizing this loss.

[0129] (6) Backpropagation algorithm

[0130] Neural networks can use the backpropagation algorithm to correct the initial neural network parameters during training, reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates error loss. This error loss information is then backpropagated to update the initial neural network parameters, thereby converging the error loss. The backpropagation algorithm is an error-driven backward propagation process aimed at obtaining the optimal neural network parameters.

[0131] In combination with the above description, the specific implementation process of the reasoning stage and the training stage of the point cloud data processing method provided in the embodiment of the present application is described below.

[0132] 1. Reasoning Stage

[0133] In the embodiment of the present application, the inference stage describes how the execution device 210 uses the target model / rule 201 to perform target detection on the collected point cloud data to generate a prediction result. For details, please refer to Figure 3 , Figure 3 Another flowchart of the method for processing point cloud data provided in an embodiment of the present application is shown below. Figure 3 First, the application of the embodiment of the present application to the field of autonomous driving is used as an example for explanation. That is, the "point cloud data corresponding to the target environment" is specifically expressed as "to-be-processed point cloud data corresponding to the surrounding environment of the target vehicle" as an example for explanation. The point cloud data processing method provided by the embodiment of the present application may include:

[0134] 301. An execution device obtains to-be-processed point cloud data corresponding to the surrounding environment of a target vehicle.

[0135] In the embodiment of the present application, the execution device needs to obtain the to-be-processed point cloud data corresponding to the surrounding environment of the target vehicle. Specifically, in some application scenarios, the execution device may be the target vehicle, and the target vehicle may directly collect the to-be-processed point cloud data corresponding to the surrounding environment of the target vehicle through sensors, including but not limited to laser radar, millimeter wave radar, or other sensors for collecting point cloud data.

[0136] Furthermore, the point cloud data to be processed may be point cloud data collected by one sensor; or may be point cloud data collected by multiple sensors and fused together.

[0137] In other application scenarios, the execution device may receive unprocessed point cloud data sent by other execution devices. That is, the unprocessed point cloud data may not necessarily describe the environment around the execution device.

[0138] In other application scenarios, the execution device may also obtain the point cloud data to be processed from multiple point cloud data stored locally. The method in which the execution device obtains the point cloud data to be processed is not limited here.

[0139] The point cloud data to be processed is used to describe the surrounding environment of the target vehicle at the target time, which can be any time. The point cloud data to be processed includes the initial information of each target point among multiple target points corresponding to the surrounding environment at the target time. The initial information of the target point includes at least the position information of the target point. Optionally, the point cloud data to be processed may also include color information, timestamp, reflection intensity (Intensity), secondary reflection intensity, or other types of initial information corresponding to each target point. The specific information of each target point included in the point cloud data to be processed needs to be determined in combination with the actual product form and is not limited here.

[0140] Furthermore, the position information of each target point can be a three-dimensional space coordinate. The aforementioned three-dimensional space coordinate can be a coordinate in a three-dimensional rectangular coordinate system, a coordinate in a spherical coordinate system, or a coordinate in another type of three-dimensional coordinate system. The position of any target point in the spherical coordinate system can be represented by three indicators: radial distance, zenith angle, and azimuth angle.

[0141] To further understand the data form of the point cloud data to be processed, as an example, for example, the execution device collects the surrounding environment data at a certain scanning rate through the laser radar to obtain the point cloud data to be processed. The point cloud data to be processed includes the coordinates of each target point in the three-dimensional rectangular coordinate system among N target points. The point cloud data to be processed can be expressed as {p j}, j = 1 to N, N represents the number of target points included in the point cloud data to be processed; p j represents any target point included in the point cloud data to be processed, p j =[x, y, z].

[0142] 302. The execution device divides the point cloud data to be processed into multiple target cubes, each target cube having S target points.

[0143] In an embodiment of the present application, after acquiring the point cloud data to be processed, the execution device may input the point cloud data to the target model. The target model then divides the point cloud data into a plurality of target cubes based on the position information of each target point included in the point cloud data. Each target cube contains S target points, where S is an integer greater than or equal to 0. Each target cube may also be referred to as a voxel, which is short for volume pixel.

[0144] For a more intuitive understanding of this solution, please refer to Figure 4 , Figure 4 A schematic diagram of the point cloud data to be processed and the target cube in the point cloud data processing method provided in an embodiment of the present application. Figure 4 In the example, the points in the point cloud data to be processed adopt the three-dimensional rectangular coordinate system. Figure 4 The figure shows point cloud data in a three-dimensional rectangular coordinate system and multiple target cubes corresponding to the point cloud data. Figure 4 In this example, only non-empty cubes are shown. Figure 4 As shown, each non-empty cube includes multiple points. It should be understood that in actual situations, the multiple target cubes corresponding to the point cloud data to be processed also include empty cubes; in addition, the points in the point cloud data to be processed can also be coordinates in other types of coordinate systems, and the specific situation should be determined in combination with the actual situation. Figure 4 The examples are only for facilitating understanding of the correspondence between the point cloud data to be processed and the target cube, and are not intended to limit this solution.

[0145] Optionally, the execution device may also set a first threshold value, which is the upper limit of the number of target points that each target cube can accommodate. If the number of target points in a target cube is greater than the first threshold value, at least one target point in the target cube that exceeds the first threshold value is discarded to ensure that the number of target points in each target cube after processing is less than or equal to the first threshold value.

[0146] 303. The execution device generates initial features of each target cube according to the initial information of the target points in each target cube.

[0147] In an embodiment of the present application, after the execution device divides the data to be processed into multiple target cubes, for any target cube within the multiple target cubes (hereinafter referred to as the "first cube" for the convenience of description), the execution device generates the initial features of the first cube and the coordinates of the first cube in the three-dimensional coordinate system through the target model based on the initial information of the target point within the first cube.

[0148] The initial feature of the first cube can be specifically represented by a vector of length d. That is, regardless of the number of target points within a target cube, the initial feature of each target cube can be represented by a vector of length d. The coordinates of the first cube in the three-dimensional coordinate system can be the coordinates of the center point of the first cube in the three-dimensional coordinate system.

[0149] Specifically, the execution device may determine whether the number of target points in the first cube is zero using the target model. If the number of target points in the first cube is not zero, the execution device may extract features of the first cube using the target model based on initial information of at least one target point in the first cube to obtain initial features of the first cube. If the number of target points in the first cube is zero, the preset information may be determined as the initial features of the first cube.

[0150] The preset information is also represented as a vector of length d. As an example, the preset information can be represented as a vector including d zeros, or as a vector including d ones, or as a vector including d different numbers, etc. It should be understood that the examples here are only for the convenience of understanding the concept of preset information. The specific form of the preset information needs to be determined in combination with actual application scenarios and is not limited here.

[0151] The execution device repeats the above steps multiple times until the initial features of each target cube corresponding to the point cloud data to be processed are obtained.

[0152] 304. The execution device updates the initial features of the multiple target cubes according to the attention mechanism to obtain updated features of the multiple target cubes.

[0153] In an embodiment of the present application, the execution device updates the initial features of multiple target cubes according to the attention mechanism to obtain updated features of the multiple target cubes. For any target cube within the multiple target cubes (hereinafter referred to as the "first cube" for the convenience of description), the execution device updates the features of the first cube according to the attention mechanism based on the initial features of the first cube through the target model to obtain the updated features of the first cube. The specific expression form of the updated features of the first cube can be similar to the specific expression form of the initial features of the first cube, and the updated features of the first cube can also be specifically expressed as a vector with a length of d.

[0154] Specifically, in one implementation, the target model may include P groups of sequentially arranged first feature update modules, where P is an integer greater than or equal to 1. This means that the execution device can update not only the initial features of the first cube but also the updated features of the first cube. A group of first feature update modules may include one first feature update module, or a group of first feature update modules may include multiple first feature modules arranged in parallel.

[0155] Step 304 may include: if the execution device determines through the first feature update module that the first cube satisfies a first preset condition, then, through the first feature update module, updating the features of the first cube according to the attention mechanism; wherein the first preset condition is that the first cube is a non-empty cube, that is, the first preset condition is that the first cube is not an empty cube, and further, a non-empty cube has the opposite meaning of an empty cube, and the features of each empty cube are preset information, that is, the features of each non-empty cube are not preset information. If the execution device determines through the first feature update module that the first cube does not satisfy the first preset condition, then, the execution device does not update the features of the first cube and processes the next target cube.

[0156] Regarding the process of determining whether the first cube satisfies the first preset condition by the execution device, in one implementation, the execution device may use a hash table to record which cubes among the multiple target cubes are empty cubes, and use a hash table to record which cubes among the multiple target cubes, that is, use a hash table to record which cubes among the multiple target cubes are non-empty cubes.

[0157] Optionally, because the execution device uses a hash table to record which cubes among the multiple target cubes are empty and which are non-empty, and the features of the empty cubes are all preset information, the feature information corresponding to the to-be-processed point cloud data may only store the features of the non-empty cubes among the multiple target cubes. Furthermore, the execution device may use sparse matrix data to store the features of the non-empty cubes.

[0158] It should be noted that, since in the process of updating the features of multiple target cubes through the first feature update module, when it is determined that the first cube is an empty cube, the features of the first cube will no longer be updated, then after the first feature update module updates the features of multiple target cubes corresponding to the point cloud data to be processed, the empty cube is still an empty cube, then after the first feature update module is used to update the features of all target cubes, the hash tables corresponding to the multiple target cubes will not change, that is, two adjacent first feature update modules can share the same hash table, so as to further reduce the amount of calculation in the process of obtaining the prediction results.

[0159] The execution device can then determine whether the first cube is an empty cube by looking up a table (i.e., a hash table) based on the index information of the first cube. The index information of each target cube can also be called the identification information of each target cube, or the global index of each target cube.

[0160] In one implementation, the index information of the first cube may be the location information of the first cube. Figure 4 As shown, multiple target cubes corresponding to the point cloud data to be processed can form a large cube, and the position information of the first cube is located by numbers in the length, width, and height dimensions respectively. As an example, for example, multiple target cubes corresponding to the point cloud data to be processed can form a large cube with a length of 68 target cubes, a width of 50 target cubes, and a height of 25 target cubes. Then, the index information of a first cube can be any one of (0, 0, 0), (0, 1, 0), ..., (31, 44, 24), (31, 45, 24), ..., or (67, 49, 24).

[0161] In another implementation, the index information of the first cube can be a character string that corresponds one-to-one to the position information of the first cube. As an example, if the position information of the first cube is (1, 2, 0), and the large cube composed of all cubes corresponding to the point cloud data to be processed is 1024×1024×42, then the index information of the first cube can be converted into 1×(1024×42)+2×42+0=43092. It should be noted that the example here is only for the convenience of understanding this solution and is not used to limit this solution.

[0162] The hash table corresponding to multiple target cubes can include two parts: a key and a value. The key part of the hash table can be obtained based on the index information of the target cube. If a target cube is an empty cube, the value corresponding to the empty target cube can be a preset value. If a target cube is not an empty cube, the value corresponding to the non-empty cube can be the order of the target cube among all non-empty cubes.

[0163] Furthermore, for the process of creating a hash table corresponding to multiple target cubes. As an example, the execution device can first establish a hash table with all values ​​of -1 (that is, taking the preset value of -1 as an example) according to the mapping range of the hash function. For all non-empty cubes, the order is the 0th non-empty cube (that is, an example of the target cube), the index information of the aforementioned target cube is (1, 2, 0), and the target cube is in the large cube composed of all cubes corresponding to the point cloud data to be processed, which is 1024×1024×42. The index information of the target cube can be converted into 1×(1024×42)+2×42+0=43092. Assuming that the hash function used is f(x)=x%1001, the keyword and value corresponding to the target cube are stored in the (43092%1001=)49th position in the hash table. The key-value pair stored in the 49th position of the hash table is (43092, 0). If a hash collision occurs, the corresponding key-value pair is postponed to the first position that is not -1. The execution device performs the above operation on all non-empty cubes, thereby completing the establishment of hash tables corresponding to multiple target cubes. It should be understood that the examples here are only for understanding the feasibility of this solution and are not used to limit this solution.

[0164] Specifically, for the query process of the first cube, the execution device obtains the location information of the first cube and obtains a character string corresponding to the location information of the first cube, inputs the character string into a hash function, and obtains the index information of the first cube in the hash table.

[0165] The execution device obtains a keyword from the corresponding position of the hash table based on the index information of the first cube in the hash table, and determines whether the obtained keyword is consistent with the obtained index information. If the judgment result is no, the execution device searches for the keyword at the next position until a keyword consistent with the obtained index information is found in the hash table, and obtains a value corresponding to the keyword from the hash table; if the judgment result is yes, the execution device obtains the value corresponding to the keyword.

[0166] Determine whether the obtained value corresponding to the keyword is a preset value. If the judgment result is yes, determine that the first cube is an empty cube; if the judgment result is no, the obtained value is the order of the first cube in multiple non-empty cubes, and then obtain the features of the first cube from the features of all non-empty cubes corresponding to the point cloud data to be processed.

[0167] In another implementation, the execution device may also obtain the features of the first cube from the feature information corresponding to all target cubes based on the index information of the first cube, and determine whether the features of the first cube are preset features. If the features of the first cube are preset features, the first cube is determined to be an empty cube; if the features of the first cube are not preset features, the first cube is determined to be a non-empty cube, and the features of the first cube are obtained.

[0168] The execution device updates the features of the first cube according to the attention mechanism by using at least one attention operator in the first feature update module to update the features of the first cube according to the attention mechanism to obtain updated features of the first cube.

[0169] Specifically, if at least one attention operator includes multiple attention operators, the execution device may update the features of the first cube according to the attention mechanism using the multiple attention operators in the first feature update module to obtain multiple updated features of the first cube. The execution device may fuse the multiple updated features of the first cube through the first feature update module to obtain the updated features of the first cube ultimately generated by the first feature update module. The at least one attention operator may include any one or more of the following: a first local attention operator, a second local attention operator, a global attention operator, a self-attention operator, or other types of attention operators.

[0170] More specifically, in one case, the execution device determines multiple second cubes corresponding to the first local attention operator through the first local attention operator in the first feature update module, and can filter out empty cubes from the multiple second cubes obtained by the first local attention operator according to the above-mentioned hash table, obtain the features of at least one filtered second cube, and then update the features of the first cube according to the attention mechanism. In the embodiment of the present application, the concepts of "features of the second cube" and "features of the first cube" are similar. The features of the second cube can be the initial features of the second cube or the updated features of the second cube.

[0171] At least one second cube corresponding to the first local attention operator includes all target cubes within a preset range around the first cube, and the distance between the second cubes within the preset range around the first cube and the first cube is less than or equal to a preset distance threshold, that is, each second cube corresponding to the first local attention operator can be in direct contact with the first cube or indirectly in contact with the first cube.

[0172] As an example, the value of the preset distance threshold may be 1 target cube, 2 target cubes, 3 target cubes or other values.

[0173] For a more intuitive understanding of this solution, please refer to Figure 5 , Figure 5 A schematic diagram of multiple second cubes corresponding to the first local attention operator in the method for processing point cloud data to be processed provided in an embodiment of the present application. The multiple target cubes included in the point cloud data to be processed are established in a three-dimensional coordinate system. Here, the relationship between the first cube and the second cube is explained only in the form of a two-dimensional image, and Ω(xyz) represents all the second cubes corresponding to the first cube, such as Figure 5 As shown, Ω(xyz)={(x′, y′, z′)||x′-x|≤1, |y′-y|≤1, |z′-z|≤1}, that is, at least one second cube corresponding to the first local attention operator includes all target cubes located around the first cube and in direct contact with the first cube. It should be understood that Figure 5 The examples are only for facilitating understanding of this solution and are not intended to limit this solution.

[0174] In another case, the execution device determines multiple second cubes corresponding to the second local attention operator through the second local attention operator in the first feature update module, and can filter out empty cubes from the multiple second cubes obtained by the second local attention operator according to the above-mentioned hash table to obtain the features of at least one filtered second cube, and then update the features of the first cube according to the attention mechanism. The at least one second cube corresponding to the second local attention operator is obtained by sampling multiple target cubes included in the point cloud data to be processed with the first cube as the sampling center. The aforementioned sampling process can be random sampling or sampling according to a preset rule.

[0175] For a more intuitive understanding of this solution, please refer to Figure 6 , Figure 6 A schematic diagram of multiple second cubes corresponding to the second local attention operator in the method for processing point cloud data to be processed provided in an embodiment of the present application. The multiple target cubes included in the point cloud data to be processed are established in a three-dimensional coordinate system. Here, the relationship between the first cube and the second cube is explained only in the form of a two-dimensional image, and Ω(xyz) represents all the second cubes corresponding to the first cube, such as Figure 6 As shown, Ω(xyz) performs the sampling operation of the second cube in a manner of gradually expanding the interval. The innermost layer is sampled in a manner of interval 1, the second inner layer is sampled in a manner of interval 2, the third layer is sampled in a manner of interval 3, and so on. It should be understood that Figure 6 The example in is only a schematic diagram for facilitating understanding of the multiple second cubes corresponding to the second local attention operator and is not used to limit this solution.

[0176] In an embodiment of the present application, two specific positions of at least one second cube corresponding to the first cube are provided, which improves the implementation flexibility of the present solution; and at least one second cube is obtained by sampling multiple target cubes included in the data to be processed, that is, the second cube may not be limited to the cube around the first cube, which greatly expands the receptive field of the target model, thereby improving the feature acquisition capability of the target model, which is conducive to improving the accuracy of the final prediction result.

[0177] In another case, the execution device can generate features of each third cube based on the features of multiple second cubes through the global attention operator in the first feature update module; wherein the aforementioned multiple second cubes include all target cubes corresponding to the point cloud data to be processed, and each third cube includes multiple second cubes, that is, all target cubes corresponding to the point cloud data to be processed are divided into multiple third cubes. The execution device can also filter out empty cubes from the multiple third cubes, and update the features of the first cube based on the global attention mechanism according to the features of each third cube that is not an empty cube; wherein, if a third cube is an empty cube, it means that all second cubes included in the third cube are empty cubes, and if there is at least one second cube in a third cube that is not an empty cube, it means that the entire third cube is not an empty cube.

[0178] Furthermore, with respect to a process in which an execution device generates features of a third cube based on features of multiple second cubes, the features of the third cube may be statistical values ​​of features of multiple second cubes included in the third cube, and the aforementioned statistical values ​​may specifically include any one or more of the following statistical features: maximum value, minimum value, mean, variance, or other types of statistical features; the coordinates of the third cube may be the coordinates of one of the multiple second cubes included in the third cube, and the specific second cube coordinates selected as the coordinates of the third cube may be determined based on the type of statistical features used for the features of the third cube.

[0179] To understand this solution more intuitively, the following formula shows the characteristics and coordinates of the third cube:

[0180]

[0181]

[0182] Among them, f super Represents the characteristics of a third cube, such as when N k =4, the minimum value, maximum value, mean value and variance can be selected as the features of the third cube, p super represents the coordinates of the third cube, MLP is the abbreviation of multilayer perceptron, This means that when the statistical types corresponding to the features of the third cube are different, the coordinates of the third cube may also be different. It should be understood that the examples given here are only for the convenience of understanding this solution and are not used to limit this solution.

[0183] For a more intuitive understanding of this solution, please refer to Figure 7 , Figure 7A schematic diagram of multiple second cubes corresponding to the global attention operator in the method for processing point cloud data to be processed provided in an embodiment of the present application. The multiple target cubes included in the point cloud data to be processed are established in a three-dimensional coordinate system. Here, the relationship between the first cube and the second cube is explained only in the form of a two-dimensional image, and Ω(xyz) represents all the second cubes corresponding to the first cube, such as Figure 7 As shown, Ω(xyz) includes all target cubes corresponding to the data to be processed. There are empty cubes and non-empty cubes in all target cubes corresponding to the data to be processed. B1 represents a third cube, that is, a third cube includes multiple second cubes. It should be understood that Figure 7 The example in is only a schematic diagram for facilitating understanding of the multiple second cubes corresponding to the second local attention operator and is not used to limit this solution.

[0184] In an embodiment of the present application, the multiple second cubes corresponding to the first cube include all target cubes, that is, the features of all target cubes corresponding to the point cloud data to be processed are used to update the features of the first cube, further expanding the receptive field of the target model and further improving the feature acquisition capability of the target model; in addition, a third cube includes multiple second cubes, that is, all target cubes are divided into multiple third cubes, and then the features of the first cube are updated using the features of the multiple third cubes, which is beneficial to improving the efficiency of the feature update process.

[0185] In another case, the execution device may also update the features of the first cube based on the self-attention mechanism according to the features of the first cube through the self-attention operator in the first feature update module.

[0186] In another case, the execution device may also perform feature updates on the first cube based on the self-attention mechanism according to the features of the first cube through the first local attention operator and the second local attention operator in the first feature update module. Figure 8 , Figure 8 A schematic diagram of multiple second cubes corresponding to the first local attention operator and the second local attention operator in the method for processing point cloud data to be processed provided in an embodiment of the present application. Figure 6 To understand the description Figure 8 The multiple target cubes included in the point cloud data to be processed are established in a three-dimensional coordinate system. Here, the relationship between the first cube and the second cube is explained in a two-dimensional image format, and Ω(xyz) represents all the second cubes corresponding to the first cube, as shown in the following example: Figure 8 As shown, Ω(xyz) not only includes all target cubes located around the first cube, such as Figure 5As shown, Ω(xyz)={(x′, y′, z′)||x′-x|≤2, |y′-y|≤2, |z′-z|≤2}, Ω(xyz) also includes performing a sampling operation of the second cube in a manner of gradually expanding the interval, the innermost layer is sampled in a manner of interval 1, the second inner layer is sampled in a manner of interval 2, the third layer is sampled in a manner of interval 3, and so on. It should be understood that Figure 8 The example in is only a schematic diagram for facilitating understanding of the multiple second cubes corresponding to the second local attention operator and is not used to limit this solution.

[0187] In another case, the execution device can also perform feature updates on the first cube based on the self-attention mechanism through the first local attention operator, the second local attention operator and the global attention operator in the first feature update module, that is, the first feature update module can include any one or more combinations of the following attention operators: the first local attention operator, the second local attention operator, the global attention operator, the self-attention operator or other types of attention operators.

[0188] Optionally, the first feature update module also includes one or more first neural network layers. After the execution device obtains at least one updated feature map corresponding to the point cloud data to be processed through at least one type of attention operator, it can use the first neural network layer to process the aforementioned at least one feature map, thereby obtaining an updated feature map corresponding to the point cloud data to be processed generated by the entire first feature update module.

[0189] For a more intuitive understanding of this solution, please refer to Figure 9a and Figure 9b , Figure 9a and Figure 9b Two schematic diagrams of the first feature update module in the method for processing point cloud data to be processed provided in the embodiment of the present application. Figure 9a , Figure 9a In the example, the first feature update module includes the first local attention operator, the second local attention operator and the global attention operator. Figure 9a As shown, after obtaining the feature map corresponding to the point cloud data to be processed (i.e., C1 in the figure), the execution device can respectively use the first local attention operator, the second local attention operator, and the global attention operator in the first feature update module to update the features of the first cube, and obtain three updated feature maps corresponding to the point cloud data to be processed (i.e., C2 in the figure). It should be understood that Figure 9a The example in is only an example for facilitating understanding of the first feature update module. The first feature update module may include more or fewer neural network layers. The example here is not used to limit this solution.

[0190] As an example, see Figure 9b , combined with Figure 9a To understand Figure 9b The first feature update module may further include a batch normalization layer, a ReLU layer, a feed forward layer, and a projection layer. After obtaining the three updated feature maps represented by C2, the execution device may further use the aforementioned multiple neural network layers to process the three updated feature maps corresponding to the point cloud data to be processed, and obtain an updated feature map corresponding to the point cloud data to be processed generated by the entire first feature update module. It should be understood that Figure 9b The example in the figure is only for the convenience of understanding the first feature update module. In actual situations, the first feature update module may include more or fewer neural network layers, and the first feature update module may also include other types of attention operators. The examples given here are not used to limit this solution.

[0191] Furthermore, since the difference between the first local attention operator and the second local attention operator is that they correspond to different second cubes corresponding to the first cube, and the global attention operator corresponds to the third cube, the execution device updates the features of the first cube according to the attention mechanism after determining the features of each second cube corresponding to the first cube (or the features of each third cube corresponding to the first cube).

[0192] To understand this solution more intuitively, the following formula is used to introduce the process of updating the features of the first cube. For example, the self-attention operator first calculates the linear mapping of the feature to generate the corresponding query feature:

[0193] query xyz =Linear query (f xyz ); (1)

[0194] Among them, query xyz Represents the query feature of the first cube, Linear query (f xyz ) represents a linear transformation of the features of the first cube.

[0195] key x′y′z′ =Linear key (f x′y′z′ )+E pos ; (2)

[0196] E pos=Linear pos (p xyz -p x′y′z′ ); (3)

[0197] Among them, key x′y′z′ represents the key feature of a second cube (or third cube), f x′y′z′ Represents the characteristics of a second cube (or third cube), Linear key (f x′y′z′ ) represents the linear transformation of the features of the second cube (or the features of the third cube), p xyz represents the coordinates of the first cube in the three-dimensional coordinate system, p x′y′z′ Represents the coordinates of the second cube (or third cube) in the three-dimensional coordinate system, Linear pos (p xyz -p x′y′z′ ) represents the p xyz -p x′y′z′ Perform linear conversion, Linear key 、Linear pos and Linear query Different linear transformation matrices may be used, or the same linear transformation matrix may be used, which is not limited here.

[0198] value x′y′z′ =Linear value (f x′y′z′ ); (4)

[0199] Among them, value x′y′z′ Represents the value feature of a second cube (or third cube), f x′y′z′ Represents the characteristics of a second cube (or third cube), Linear value (f x′y′z′ ) represents a linear transformation of the features of the second cube (or the features of the third cube), Linear value 、Linear key 、Linear pos and Linear query Different linear transformation matrices may be used, or the same linear transformation matrix may be used, which is not limited here.

[0200] After the execution device generates the key feature of each second cube (or third cube) through formula (2) and generates the value feature of each second cube (or third cube) through formula (4), it can generate the updated feature of the first cube through the following formula.

[0201]

[0202] in, represents the updated features of the first cube generated by an attention operator (which can be any of the first local attention operator, the second local attention operator, or the global attention operator), Ω(xyz) represents the second cube (or third cube) corresponding to the first cube, Represents key x′y′z′ The transpose of d k Linear key (f x′y′z′ ), the meanings of other elements in formula (5) can be found in the descriptions in formulas (1) to (4), which will not be repeated here.

[0203] In the embodiment of the present application, the feature update of the first cube will be performed according to the attention mechanism only when the first cube is not an empty cube. That is, after the multiple target cubes corresponding to the point cloud data to be processed are updated by the first feature update module, the distribution of hollow cubes and non-empty cubes in the multiple target cubes corresponding to the point cloud data to be processed will not change, thereby realizing the feature update of multiple target cubes and helping to retain the edge features of the object.

[0204] In another implementation, the target model may include Q second feature update modules, and step 304 may include: when the first cube meets the second preset condition, the execution device updates the features of the first cube among the multiple target cubes according to the attention mechanism through the second feature update module; when the first cube does not meet the second preset condition, the execution device continues to process the next first cube.

[0205] Among them, the second preset condition is that there is a non-empty cube in the cube set, the cube set includes a first cube and at least one second cube corresponding to the first cube, at least one second cube is determined from multiple target cubes according to the attention mechanism, and at least one second cube is determined from multiple target cubes according to the attention mechanism. That is, the second preset condition is that as long as there is a non-empty cube in the entire cube set, the second feature update module will be triggered to update the features of the current first cube.

[0206] Furthermore, similar to the first feature update module, the second feature update module includes any one or more of the following attention operators: a first local attention operator, a second local attention operator, a global attention operator, a self-attention operator or other types of attention operators; that is, the difference between the second feature update module and the first feature update module is only that the triggering condition of the first feature update module is the first preset condition, and the triggering condition of the second feature update module is the second preset condition. For the specific implementation method of "updating the features of the first cube according to the attention mechanism through the second feature update module", please refer to the above-mentioned specific implementation method of "updating the features of the first cube according to the attention mechanism through the first feature update module", which will not be repeated here.

[0207] Furthermore, regarding the process of the execution device determining whether the first cube meets the second preset condition. In one implementation, the execution device may use a hash table to record which cubes among multiple target cubes are empty cubes, and use a hash table to record which cubes among multiple target cubes, that is, use a hash table to record which cubes among multiple target cubes are non-empty cubes. The execution device may then determine whether the first cube meets the second preset condition by looking up a table (that is, a hash table) based on the index information of the first cube and the index information of all second cubes corresponding to the first cube. The specific implementation of the aforementioned steps can be found in the above description and will not be elaborated here.

[0208] It should be noted that in order to further improve the speed of the table lookup process, in each table lookup process, it is possible to determine whether it is an empty cube based on the index information of multiple second cubes. Then, the table lookup complexity corresponding to the process of executing the device to update the features of a first cube is Where m represents the number of non-empty cubes in all second cubes corresponding to the first cube, n represents the size of the hash table, and time index Represents the duration of a series of subsequent indexes. Since the duration of an index operation is fixed in the computer, no matter how many cubes need to be indexed, time index The complexity is usually at a constant level, so the method of using a hash table to record the empty cubes in the multiple target cubes can greatly improve the efficiency of the process of obtaining the features of non-empty cubes, which is conducive to improving the efficiency of the process of target detection based on point cloud data.

[0209] For a more intuitive understanding of this solution, please refer to Figure 10 , Figure 10 A schematic diagram of using a hash table to determine whether multiple target cubes are empty cubes in the method for processing point cloud data to be processed provided in an embodiment of the present application. Figure 10In the example, the index information of the target cube is the location information of the target cube. Figure 10 As shown, the position information of the first cube being processed by the execution device is (32, 45, 78), and the multiple second cubes obtained by the global attention operator and the second local attention operator include the position information of four second cubes: (31, 44, 78), (31, 46, 78), (33, 44, 78) and (33, 46, 78).

[0210] The execution device obtains the five character strings 12219, 16820, 17953, 18906 and 20145 corresponding to the five position information (31, 44, 78), (31, 46, 78), (32, 45, 78), (33, 44, 78) and (33, 46, 78) respectively, and based on the hash function, obtains the positions of the five character strings 12219, 16820, 17953, 18906 and 20145 in the hash table. By looking up the table, it can be seen that two second cubes are empty cubes.

[0211] The execution device obtains from the hash table the order of the three target cubes (31, 44, 78), (31, 46, 78) and (32, 45, 78) (i.e., two second cubes and one first cube) in the feature map as 102, 158 and 179 respectively. Then, the features of the three target cubes are obtained from the feature map corresponding to the point cloud data to be processed. It should be understood that Figure 10 The examples are only for facilitating understanding of this solution and are not intended to limit this solution.

[0212] In the embodiment of the present application, only when the first cube and multiple second cubes are empty cubes will the features of the first cube not be updated through the second feature updating module, which is conducive to updating the features of more target cubes and improving the feature acquisition capability of the target model.

[0213] In another implementation, the target model may include P first feature update modules and Q second feature update modules, where the value of P is optionally greater than the value of Q. Step 304 may include: if the first cube meets a first preset condition, the execution device may update the features of the first cube using the first feature update module according to the attention mechanism. If the first cube meets a second preset condition, the execution device may update the features of the first cube using the second feature update module according to the attention mechanism.

[0214] In an embodiment of the present application, the target model may include P first feature update modules and Q second feature update modules, and the value of P is greater than the value of Q, thereby improving the feature acquisition capability of the target model while retaining the edge features of the object as much as possible, which is conducive to improving the accuracy of the final prediction results.

[0215] For a more intuitive understanding of this solution, please refer to Figure 11 , Figure 11 A schematic diagram of a feature update module of a target model in a method for processing point cloud data to be processed provided in an embodiment of the present application. Figure 11 As shown, two first feature update modules and one second feature update module can form a group. Figure 11 Taking the target model's feature update module including three groups as an example, that is, the execution device can update the features of multiple target cubes corresponding to the point cloud data to be processed multiple times through multiple first feature update modules and multiple second feature update modules in the target model. It should be understood that Figure 11 The examples are only for facilitating understanding of this solution and are not intended to limit this solution.

[0216] 305. The execution device performs a target detection operation on the updated features of the multiple target cubes to obtain a prediction result corresponding to the point cloud data to be processed, where the prediction result is used to indicate the position of at least one object in the surrounding environment of the target vehicle.

[0217] In an embodiment of the present application, the execution device updates the features of multiple target cubes corresponding to the point cloud data to be processed through the target model according to the attention mechanism, and obtains updated features of the multiple target cubes corresponding to the point cloud data to be processed. The target detection can be performed through the target model based on the updated features of the multiple target cubes corresponding to the point cloud data to be processed, thereby obtaining a prediction result corresponding to the point cloud data to be processed, and the prediction result is used to indicate the position of at least one object in the surrounding environment of the target vehicle.

[0218] In an embodiment of the present application, the point cloud data corresponding to the surrounding environment of the target vehicle is divided into multiple cubes, and then the target detection operation is performed based on the features of the multiple target cubes. That is, the minimum granularity of the processing object in the present application is no longer a target point, but a cube, which reduces the computer resources consumed in the entire point cloud data processing process and is conducive to improving the operating efficiency of the target model; and according to the initial information of the target point in the first cube, after the initial features of the first cube are generated by the target model, the updated features of the first cube are generated according to the initial features of the first cube according to the attention mechanism, and then target detection is performed according to the updated features of the multiple target cubes, which is conducive to focusing on the important information carried by the point cloud data to improve the accuracy of the prediction results.

[0219] This application embodiment also provides another method for processing point cloud data. Figure 12 , Figure 12 This is another flow chart of a method for processing point cloud data provided in an embodiment of the present application. The method for processing point cloud data provided in an embodiment of the present application may include:

[0220] 1201. The execution device obtains the to-be-processed point cloud data corresponding to the target environment.

[0221] 1202. The execution device divides the point cloud data to be processed into multiple target cubes, each target cube having S target points.

[0222] 1203. The execution device generates initial features of each target cube according to the initial information of the target points in each target cube.

[0223] 1204. The execution device updates the initial features of the multiple target cubes according to the attention mechanism to obtain updated features of the multiple target cubes.

[0224] In the embodiment of the present application, the specific implementation of the execution device performing steps 1202 to 1204 can be referred to. Figure 3 The description of the specific implementation of steps 302 to 304 in the corresponding embodiment is not repeated here.

[0225] 1205. The execution device performs feature processing operations on the updated features of the multiple target cubes to obtain prediction results corresponding to the point cloud data.

[0226] In an embodiment of the present application, an execution device processes updated features of multiple target cubes using a target model. The prediction result indicates a prediction result corresponding to the point cloud data to be processed, and the prediction result represents information about at least one object in the target environment. Furthermore, the prediction result represents position information, category information, or other types of information about at least one object in the target environment. The target model may include any of the following functions: target detection, face recognition, or gesture recognition.

[0227] When the function of the target model is target detection, step 1205 may include: performing a target detection operation based on the updated features of the multiple target cubes to obtain a prediction result corresponding to the point cloud data, wherein the prediction result is used to indicate the position information of at least one object in the target environment. Figure 3 The specific implementation of step 305 in the corresponding embodiment will not be described here. Figure 3 In the corresponding embodiment, the execution device can be a vehicle; Figure 12In the corresponding embodiment, the execution device can be any of the following devices: a vehicle, a mobile robot, a monitoring data processing device or other devices for processing point cloud data, etc., which are not exhaustive here. Figure 3 In the corresponding embodiment, the point cloud data to be processed refers to the point cloud data corresponding to the vehicle's surrounding environment; Figure 12 In the corresponding embodiment, the point cloud data to be processed can be any of the following: point cloud data corresponding to the vehicle's surroundings, point cloud data corresponding to the mobile robot's surroundings, point cloud data collected by monitoring, or other types of point cloud data. Figure 3 In the corresponding embodiment, the prediction result is used to indicate the position of at least one object in the vehicle's surrounding environment; Figure 12 In the corresponding embodiment, the prediction result is used to indicate any of the following information: the position of at least one object in the vehicle's surrounding environment, the position of at least one object in the mobile robot's surrounding environment, or the environment of at least one object in the monitoring environment, etc., which are not exhaustive here.

[0228] When the function of the target model is face recognition, step 1205 may include: performing a face recognition operation based on the updated features of the multiple target cubes to obtain a prediction result corresponding to the point cloud data, the prediction result representing the category information of at least one object in the target environment, and the category information of at least one object in the target environment is used to indicate whether the face match is successful or failed. The execution device configured with the target model can be any of the following devices: a terminal device, a smart door lock, a monitoring data processing device, and the prediction result corresponding to the point cloud data to be processed is used to indicate whether the match is successful or failed. The specific implementation method of step 1205 will be described later. Figure 14 The corresponding embodiments are described in detail and will not be described here in detail.

[0229] When the function of the target model is posture recognition, step 1205 may include: performing a posture recognition operation based on the updated features of the multiple target cubes to obtain a prediction result corresponding to the point cloud data, wherein the information of at least one object in the target environment includes the physical features of at least one human body in the target environment and / or the gait features of at least one human body in the target environment when walking. The prediction result corresponding to the point cloud data to be processed is used to indicate the posture features corresponding to the human body. The posture features may include physical features and / or gait features when walking. The meaning of the posture features can be referred to the above description and will not be repeated here.

[0230] In the embodiments of the present application, multiple specific application scenarios of this solution are provided, which improves the implementation flexibility of this solution.

[0231] 2. Training Phase

[0232] In the embodiment of the present application, the training phase describes the process of how the training device 220 generates a mature neural network using the image data set in the database 230. For details, please refer to Figure 13 , Figure 13 A flowchart of a neural network training method provided in an embodiment of the present application is provided. The neural network training method provided in an embodiment of the present application may include:

[0233] 1301. The training device obtains the point cloud data to be processed corresponding to the target environment.

[0234] 1302. The training device divides the point cloud data to be processed into multiple target cubes, each of which contains S target points.

[0235] 1303. The training device generates initial features of each target cube through the model to be trained based on the initial information of the target points in each target cube.

[0236] 1304. The training device updates the initial features of the multiple target cubes according to the attention mechanism to obtain updated features of the multiple target cubes.

[0237] 1305. The training device performs feature processing operations on the model to be trained based on the updated features of the multiple target cubes to obtain prediction results corresponding to the point cloud data to be processed.

[0238] In the embodiment of the present application, the specific implementation of the training device performing steps 1301 to 1305 can be found in Figure 12 The description of the specific implementation of steps 301 to 305 in the corresponding embodiment is not repeated here.

[0239] 1306. The training device trains the to-be-trained model according to a target loss function, wherein the target loss function is used to indicate the similarity between the predicted result and the expected result corresponding to the point cloud data.

[0240] In an embodiment of the present application, the training device is pre-configured with training data, which includes expected results corresponding to the point cloud data to be processed. After obtaining a prediction result corresponding to the point cloud data to be processed, the training device can calculate the function value of a target loss function based on the prediction result corresponding to the point cloud data to be processed and the expected result corresponding to the point cloud data to be processed. The parameters of the model to be trained are updated based on the function value of the target loss function and a backpropagation algorithm, thereby completing a training of the model to be trained.

[0241] Among them, the "model to be trained" can also be understood as the "target model to be trained". The meaning represented by the "expected result corresponding to the point cloud data to be processed" is similar to the meaning of the "prediction result corresponding to the point cloud data to be processed", the difference being that the "prediction result corresponding to the point cloud data to be processed" is the prediction result generated by the model to be trained, and the "expected result corresponding to the point cloud data to be processed" is the correct result corresponding to the point cloud data to be processed. As an example, when the model to be processed is used to perform a target detection task, the prediction result is used to indicate the expected position of at least one object in the target environment, and the expected result is used to indicate the expected position (also referred to as the correct position) of at least one object in the target environment. It should be understood that the examples given here are only for the convenience of understanding this solution, and are not intended to exhaustively enumerate the meanings of the expected results in various application scenarios.

[0242] The training device can repeat steps 1301 to 1306 multiple times to achieve iterative training of the model to be trained until the preset conditions are met and the trained model to be trained is obtained, wherein the preset conditions can be the convergence conditions of the target loss function, or the number of iterations of steps 1301 to 1306 reaches a preset number.

[0243] In the embodiments of the present application, not only a specific implementation method of the model's reasoning process is provided, but also a specific implementation method of the model's training process is provided, thereby expanding the application scenarios of this solution.

[0244] This application embodiment also provides a method for processing point cloud data. Figure 14 , Figure 14 A flowchart of a method for processing point cloud data provided in an embodiment of the present application is provided. The method for processing point cloud data provided in an embodiment of the present application may include:

[0245] 1401. The execution device detects an operation indicating face recognition.

[0246] In an embodiment of the present application, the executing device may detect an operation indicating facial recognition. Specifically, in one application scenario, if the executing device is a terminal device such as a mobile phone or tablet, the executing device is deemed to have detected an operation indicating facial recognition when it detects any of the following operations: unlocking the screen, making a payment, opening encrypted data, or other operations requiring identity recognition, which are not exhaustive here.

[0247] In another application scenario, if the executing device is a smart door lock, when the executing device detects a door opening operation input by the user, it can be regarded as detecting an operation indicating face recognition.

[0248] In another application scenario, if the execution device is a device for processing monitoring data, when the execution device detects a face matching operation input by the user, it can be regarded as detecting an operation indicating face recognition.

[0249] In the embodiments of the present application, multiple application scenarios for detecting face recognition operations are provided, which improves the implementation flexibility of the present solution.

[0250] 1402. The execution device outputs a face recognition result in response to an operation indicating face recognition, where the face recognition result includes a successful match and a failed match. The face recognition result is obtained based on updated features of multiple target cubes, where the updated features of the target cube are obtained by updating the initial features of the target cube according to an attention mechanism, where the initial features of the target cube are obtained based on collected point cloud data corresponding to the target environment, where the point cloud data includes initial information of multiple target points, and the point cloud data is divided into multiple target cubes, where there are S target points in one target cube, where S is an integer greater than or equal to 0.

[0251] In an embodiment of the present application, the execution device, in response to an operation instructing facial recognition, can trigger acquisition of point cloud data corresponding to the target environment. The point cloud data includes initial information about multiple target points, and the point cloud data is divided into multiple target cubes. Each target cube contains S target points, where S is an integer greater than or equal to 0.

[0252] Furthermore, the execution device can trigger the collection of point cloud data corresponding to the surrounding environment in response to an operation indicating face recognition; or, the execution device pre-stores point cloud data corresponding to the target environment collected by the monitoring device, and the execution device can trigger the acquisition of point cloud data corresponding to the target environment in response to an operation indicating face recognition.

[0253] The execution device generates the initial features of the first cube using the target model based on the initial information of the target point in the first cube, and updates the features of the first cube using the target model according to the initial features of the first cube and the attention mechanism to obtain the updated features of the first cube. The first cube is one of the multiple target cubes. It should be noted that the specific implementation method of the execution device obtaining the updated features of the first cube based on the acquired point cloud data can be found in Figure 3 The description in the corresponding embodiment is not repeated here.

[0254] The execution device performs a face recognition operation through a target model according to the updated features of the multiple target cubes, and outputs a face recognition result, which includes a successful match and a failed match.

[0255] Specifically, the facial features of at least one user may be pre-stored on the execution device. The execution device may match the updated features of multiple target cubes with the facial features of each of the aforementioned users to obtain a face recognition result, which includes a successful match and a failed match.

[0256] Specifically, for example, if the face recognition operation is triggered based on the operation of unlocking the screen, when the face recognition result is a successful match, the execution device can display the interface after the screen is unlocked, or can display how to enter the interface after the screen is unlocked, etc.; when the face recognition result is a failed match, the execution device can display the unlocking failure interface.

[0257] For example, if the facial recognition operation is triggered based on a payment operation, when the facial recognition result is a successful match, the execution device can trigger the payment operation; when the facial recognition result is a failed match, the execution device can stop the payment.

[0258] For example, if the face recognition operation is triggered based on the operation of opening encrypted data, when the face recognition result is a successful match, the execution device can trigger the display of the encrypted data; when the face recognition result is a failed match, the execution device can display the "Failed to open encrypted data" interface.

[0259] For example, if the facial recognition operation is triggered based on the door opening operation, when the facial recognition result is a successful match, the execution device can trigger the door to open; when the facial recognition result is a failed match, the execution device may not open the door and may issue a warning sound.

[0260] For example, if the face recognition operation is triggered based on the face matching operation, when the face recognition result is a successful match, the execution device can display the personal information of the matched person; when the face recognition result is a failed match, the execution device can inform the user of the match failure through a text box, voice or other means.

[0261] For a more intuitive understanding of this solution, please refer to Figure 15 , Figure 15 A schematic diagram of an operation indicating face recognition detected in the point cloud data processing method provided in an embodiment of the present application. Figure 15 Including two sub-schematic diagrams on the left and right, Figure 15 Take the application scenario of unlocking the screen as an example. Figure 15 As shown in the left diagram, the execution device is performing face recognition; if the face recognition result is a successful match, it will trigger the entry Figure 15 The diagram on the right shows how to enter the interface after the screen is unlocked. You should understand that Figure 15 The examples are only for facilitating understanding of this solution and are not intended to limit this solution.

[0262] To provide a more intuitive understanding of the beneficial effects of the embodiments of this application, the following is an explanation using specific experimental data. Experiments were conducted using the KITTI dataset and the Waymo dataset. The KITTI dataset contains approximately 13,000 frames of point cloud scenes, of which approximately 3,500 frames are divided into a training dataset, approximately 3,500 frames are divided into a validation dataset, and the remaining 7,000 frames are divided into a test dataset. For the Waymo dataset, we randomly selected 20% of the data in the Waymo dataset as the training dataset and used the entire Waymo dataset as the validation dataset.

[0263] The experiment used mAP as a metric to measure 3D object detection performance. On the KITTI dataset, the model in this embodiment of the application improved detection performance by 4% compared to the Second model, and by 2% compared to the PV-RCNN model.

[0264] On the Waymo dataset, the model in this application's example improved detection performance by 1% compared to the Second model and by 3% compared to the PV-RCNN model. The experiment also tested the number of parameters and inference speed. The experimental data can be found in the table below.

[0265] Model Inference speed (frames / second) Second 20.73 Models in the embodiments of this application 13.65

[0266] Table 1

[0267] Table 1 takes the KITTI dataset as an example for the experiment. As shown in Table 1, the target model provided by the embodiment of the present application can also improve the data processing efficiency.

[0268] Model Parameter quantity Second 5.3M Models in the embodiments of this application 4.8M

[0269] Table 2

[0270] As shown in Table 2, the number of parameters of the target model provided by the embodiment of the present application is smaller.

[0271] In Figures 1 to Figure 13 On the basis of the corresponding embodiment, in order to better implement the above solution of the embodiment of the present application, the following also provides related equipment for implementing the above solution. Figure 16 , Figure 16A structural schematic diagram of a point cloud data processing device provided in an embodiment of the present application, the point cloud data processing device 1600 may include: an acquisition module 1601, used to acquire point cloud data corresponding to a target environment, wherein the point cloud data includes initial information of multiple target points, and the point cloud data is divided into multiple target cubes, each target cube has S target points, and S is an integer greater than or equal to 0; a generation module 1602, used to generate initial features of each target cube based on the initial information of the target points in each target cube; an update module 1603, used to update the initial features of the multiple target cubes according to the attention mechanism to obtain updated features of the multiple target cubes; a processing module 1604, used to perform feature processing operations on the updated features of the multiple target cubes to obtain prediction results corresponding to the point cloud data, and the prediction results represent information of at least one object in the target environment.

[0272] In one possible design, the update module 1603 is specifically used to update the features of each first cube in multiple target cubes according to the attention mechanism when the first cube meets the first preset condition. The first cube is any cube in the multiple target cubes, the first preset condition is a non-empty cube, and the feature of the non-empty cube is not preset information.

[0273] In one possible design, the apparatus is applied to an execution device, and the execution device uses a hash table to record non-empty cubes in multiple target cubes.

[0274] In one possible design, the update module 1603 is further used to update the features of each first cube in the multiple target cubes according to the attention mechanism when the first cube meets the second preset condition; wherein the second preset condition is that there is a non-empty cube in the cube set, the cube set includes the first cube and at least one second cube corresponding to the first cube, and the at least one second cube is determined from the multiple target cubes according to the attention mechanism.

[0275] In one possible design, at least one second cube includes all target cubes within a preset range around the first cube, the distance between the second cubes within the preset range around the first cube and the first cube is less than or equal to a preset distance threshold, and / or, at least one second cube is obtained by sampling multiple target cubes with the first cube as the sampling center.

[0276] In one possible design, the update module 1603 is further used to: generate features of multiple third cubes based on the features of the second cube, where one third cube includes multiple second cubes; and update the features of each first cube in the multiple target cubes based on the global attention mechanism based on the features of each third cube, where the global attention mechanism is one type of attention mechanism.

[0277] In one possible design, the point cloud data corresponding to the target environment includes point cloud data corresponding to the environment surrounding the target vehicle, and the processing module 1604 is specifically used to: perform a target detection operation on the updated features of the target cube to obtain a prediction result corresponding to the point cloud data, and the prediction result is used to indicate the position of at least one object in the environment surrounding the target vehicle.

[0278] In one possible design, the processing module 1604 is specifically used to: perform a target detection operation based on the updated features of multiple target cubes to obtain a prediction result corresponding to the point cloud data, and the information of at least one object in the target environment includes the position information of at least one object in the target environment; or, perform a face recognition operation based on the updated features of multiple target cubes to obtain a prediction result corresponding to the point cloud data, and the information of at least one object in the target environment includes the category information of at least one object in the target environment, and the category information of at least one object in the target environment is used to indicate whether the face match is successful or failed; or, perform a posture recognition operation based on the updated features of multiple target cubes to obtain a prediction result corresponding to the point cloud data, and the information of at least one object in the target environment includes the physical features of at least one human body in the target environment and / or the gait features of at least one human body when walking in the target environment.

[0279] It should be noted that the information interaction, execution process, etc. between the modules / units in the point cloud data processing device 1600 are the same as those in the present application. Figure 12 The corresponding method embodiments are based on the same concept. For specific contents, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0280] The present application also provides a point cloud data processing device. Figure 17 , Figure 17A structural schematic diagram of a point cloud data processing device provided in an embodiment of the present application, the point cloud data processing device 1700 may include: an acquisition module 1701, used to acquire point cloud data corresponding to the surrounding environment of a target vehicle, wherein the point cloud data includes initial information of multiple target points, and the point cloud data is divided into multiple target cubes, each target cube has S target points, and S is an integer greater than or equal to 0; a generation module 1702, used to generate initial features of each target cube based on the initial information of the target points in each target cube; an update module 1703, used to update the initial features of the multiple target cubes according to the attention mechanism to obtain updated features of the multiple target cubes; a processing module 1704, used to perform feature processing operations on the updated features of the multiple target cubes to obtain prediction results corresponding to the point cloud data, and the prediction results are used to indicate the position of at least one object in the surrounding environment of the target vehicle.

[0281] In one possible design, the update module 1703 is specifically used to update the features of each first cube in multiple target cubes according to the attention mechanism when the first cube meets the first preset condition. The first cube is any cube in the multiple target cubes, the first preset condition is a non-empty cube, and the feature of the non-empty cube is not preset information.

[0282] In one possible design, the update module 1703 is further used to update the features of each first cube in the multiple target cubes according to the attention mechanism when the first cube meets the second preset condition; wherein the second preset condition is that there is a non-empty cube in the cube set, the cube set includes the first cube and at least one second cube corresponding to the first cube, and the at least one second cube is determined from the multiple target cubes according to the attention mechanism.

[0283] It should be noted that the information interaction, execution process, etc. between the modules / units in the point cloud data processing device 1700 are the same as those in the present application. Figures 2b to 11 The corresponding method embodiments are based on the same concept. For specific contents, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0284] The present application also provides a neural network training device, see Figure 18 , Figure 18A structural schematic diagram of a neural network training device provided in an embodiment of the present application, the neural network training device 1800 may include: an acquisition module 1801, used to acquire point cloud data corresponding to a target environment, wherein the point cloud data includes initial information of multiple target points, and the point cloud data is divided into multiple target cubes, each target cube has S target points, and S is an integer greater than or equal to 0; a generation module 1802, used to generate initial features of each target cube based on the initial information of the target points in each target cube; an update module 1803, used to update the initial features of each target cube in the initial features of the multiple target cubes according to the attention mechanism to obtain updated features of the multiple target cubes; a processing module 1804, used to perform feature processing operations on the updated features of the multiple target cubes to obtain prediction results corresponding to the point cloud data; a training module 1805, used to train the target model according to a target loss function, wherein the target loss function is used to indicate the similarity between the prediction result and the expected result corresponding to the point cloud data.

[0285] In one possible design, the update module 1803 is specifically used to update the features of each first cube in multiple target cubes according to the attention mechanism when the first cube meets the first preset condition. The first cube is any cube in the multiple target cubes, the first preset condition is a non-empty cube, and the feature of the non-empty cube is not preset information.

[0286] In one possible design, the update module 1803 is further used to update the features of each first cube in the multiple target cubes according to the attention mechanism when the first cube meets the second preset condition; wherein the second preset condition is that there is a non-empty cube in the cube set, the cube set includes the first cube and at least one second cube corresponding to the first cube, and the at least one second cube is determined from the multiple target cubes according to the attention mechanism.

[0287] It should be noted that the information interaction, execution process, etc. between the modules / units in the neural network training device 1800 are the same as those in the present application. Figure 13 The corresponding method embodiments are based on the same concept. For specific contents, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0288] The present application also provides another point cloud data processing device. Figure 19 , Figure 19Another structural schematic diagram of a point cloud data processing device provided in an embodiment of the present application, the point cloud data processing device 1900 may include: a detection module 1901, for detecting an operation indicating face recognition; an output module 1902, for outputting a face recognition result in response to an operation indicating face recognition; wherein the face recognition result includes a successful match and a failed match, and the face recognition result is obtained based on the updated features of multiple target cubes, the updated features of the target cube are obtained by updating the initial features of the target cube according to the attention mechanism, the initial features of the target cube are obtained based on the collected point cloud data corresponding to the target environment, the point cloud data includes initial information of multiple target points, the point cloud data is divided into multiple target cubes, there are S target points in a target cube, and S is an integer greater than or equal to 0.

[0289] In one possible design, when the detection module 1901 detects any of the following operations, it is considered to have detected an operation indicating face recognition: an operation to unlock the screen, a payment operation, or an operation to open encrypted data; or, a door opening operation input by the user is detected; or, a face matching operation input by the user is detected.

[0290] It should be noted that the information interaction, execution process, etc. between the modules / units in the point cloud data processing device 1900 are the same as those in the present application. Figure 14 The corresponding method embodiments are based on the same concept. For specific contents, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0291] Next, we will introduce an execution device provided by the embodiment of the present application. Figure 20 , Figure 20 This is a schematic diagram of the structure of an execution device provided in an embodiment of the present application. The execution device 2000 can be specifically manifested as a vehicle, a mobile robot, a monitoring data processing device, or other devices, etc., which are not limited here. Specifically, the execution device 2000 includes: a receiver 2001, a transmitter 2002, a processor 2003, and a memory 2004 (wherein the number of processors 2003 in the execution device 2000 can be one or more, Figure 20 (taking one processor as an example), the processor 2003 may include an application processor 20031 and a communication processor 20032. In some embodiments of the present application, the receiver 2001, the transmitter 2002, the processor 2003 and the memory 2004 may be connected via a bus or other means.

[0292] The memory 2004 may include a read-only memory and a random access memory, and provides instructions and data to the processor 2003. A portion of the memory 2004 may also include non-volatile random access memory (NVRAM). The memory 2004 stores processor and operation instructions, executable modules, or data structures, or subsets or extended sets thereof. The operation instructions may include various operation instructions for implementing various operations.

[0293] Processor 2003 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together via a bus system. In addition to a data bus, the bus system may also include a power bus, a control bus, and a status signal bus. However, for clarity, all of these buses are referred to as a bus system in the figure.

[0294] The methods disclosed in the above embodiments of the present application can be applied to or implemented by processor 2003. Processor 2003 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in processor 2003 or software instructions. The above processor 2003 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and can further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 2003 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of the present application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in memory 2004, and processor 2003 reads information in memory 2004 and, in conjunction with its hardware, completes the steps of the above method.

[0295] Receiver 2001 can be used to receive input digital or character information and generate signal input related to executing device-related settings and function control. Transmitter 2002 can be used to output digital or character information through the first interface. Transmitter 2002 can also be used to send instructions to the disk pack through the first interface to modify data in the disk pack. Transmitter 2002 can also include a display device such as a display screen.

[0296] In one embodiment of the present application, the application processor 20031 in the processor 2003 is configured to execute Figures 2b to 11 The point cloud data processing method executed by the execution device in the corresponding embodiment. It should be noted that the specific manner in which the application processor 20031 executes the above steps is the same as that in the present application. Figures 2b to 11 The corresponding method embodiments are based on the same concept, and the technical effects they bring are the same as those in this application. Figures 2b to 11 The corresponding method embodiments are the same. For specific contents, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0297] In another embodiment, the application processor 20031 in the processor 2003 is used to execute Figure 12 The point cloud data processing method executed by the execution device in the corresponding embodiment. It should be noted that the specific manner in which the application processor 20031 executes the above steps is the same as that in the present application. Figure 12 The corresponding method embodiments are based on the same concept, and the technical effects they bring are the same as those in this application. Figure 12 The corresponding method embodiments are the same. For specific contents, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0298] In another embodiment, the application processor 20031 in the processor 2003 is used to execute Figure 12 The point cloud data processing method executed by the execution device in the corresponding embodiment. It should be noted that the specific manner in which the application processor 20031 executes the above steps is the same as that in the present application. Figure 14 The corresponding method embodiments are based on the same concept, and the technical effects they bring are the same as those in this application. Figure 14 The corresponding method embodiments are the same. For specific contents, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0299] The present application also provides a training device. Figure 21 , Figure 21This is a structural diagram of a training device provided in an embodiment of the present application. Specifically, the training device 2100 is implemented by one or more servers. The training device 2100 may have relatively large differences due to different configurations or performance. It may include one or more central processing units (CPUs) 2122 (for example, one or more processors) and memory 2132, and one or more storage media 2130 (for example, one or more mass storage devices) storing application programs 2142 or data 2144. Among them, the memory 2132 and the storage medium 2130 can be temporary storage or permanent storage. The program stored in the storage medium 2130 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the training device. Furthermore, the central processing unit 2122 can be configured to communicate with the storage medium 2130 to execute a series of instruction operations in the storage medium 2130 on the training device 2100.

[0300] The training device 2100 may also include one or more power supplies 2126, one or more wired or wireless network interfaces 2150, one or more input and output interfaces 2158, and / or one or more operating systems 2141, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0301] In the embodiment of the present application, the central processing unit 2122 is used to execute Figure 13 The point cloud data processing method executed by the training device in the corresponding embodiment. It should be noted that the specific manner in which the central processor 2122 executes the above steps is the same as that in the present application. Figure 13 The corresponding method embodiments are based on the same concept, and the technical effects they bring are the same as those in this application. Figure 13 The corresponding method embodiments are the same. For specific contents, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0302] The present application also provides a computer program product which, when executed on a computer, enables the computer to execute the aforementioned Figures 2b to 11 The steps executed by the execution device in the method described in the embodiment shown, or the computer is caused to execute the steps as described above Figure 12 The steps executed by the execution device in the method described in the embodiment shown, or the computer is caused to execute the steps as described above Figure 13 The steps performed by the training device in the method described in the embodiment shown, or the steps of making the computer perform the above Figure 14 or Figure 15 The illustrated embodiments describe the steps performed by the device in the method.

[0303] The present application also provides a computer-readable storage medium in which a program for signal processing is stored. When the program is run on a computer, the computer executes the above-mentioned Figures 2b to 11 The steps executed by the execution device in the method described in the embodiment shown, or the computer is caused to execute the steps as described above Figure 12 The steps executed by the execution device in the method described in the embodiment shown, or the computer is caused to execute the steps as described above Figure 13 The steps performed by the training device in the method described in the embodiment shown, or the steps of making the computer perform the above Figure 14 or Figure 15 The illustrated embodiments describe the steps performed by the device in the method.

[0304] The point cloud data processing device, neural network training device, execution device and training device provided in the embodiment of the present application can be specifically a chip, which includes: a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, a pin or a circuit. The processing unit can execute the computer execution instructions stored in the storage unit to enable the chip to execute the above Figures 2b to 11 The method for processing point cloud data described in the embodiment shown, or to enable the chip to perform the above Figure 12 The method for processing point cloud data described in the embodiment shown, or to enable the chip to perform the above Figure 13 The neural network training method described in the embodiment shown, or to enable the chip to perform the above Figure 14 or Figure 15 The method for processing point cloud data described in the illustrated embodiment. Optionally, the storage unit is a storage unit within the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM), etc.

[0305] For details, please refer to Figure 22 , Figure 22 This is a schematic diagram of the structure of a chip provided in an embodiment of the present application. The chip can be represented as a neural network processor NPU 220. NPU 220 is mounted on the host CPU as a coprocessor and is assigned tasks by the host CPU. The core of the NPU is arithmetic circuit 2203, which is controlled by controller 2204 to extract matrix data from memory and perform multiplication operations.

[0306] In some implementations, the arithmetic circuit 2203 includes multiple processing units (PEs). In some implementations, the arithmetic circuit 2203 is a two-dimensional systolic array. The arithmetic circuit 2203 can also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 2203 is a general-purpose matrix processor.

[0307] For example, assume there are input matrix A, weight matrix B, and output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from weight memory 2202 and caches it on each PE in the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from input memory 2201 and performs a matrix operation on matrix B. The partial or final matrix result is stored in accumulator 2208.

[0308] Unified memory 2206 is used to store input and output data. Weight data is directly transferred to weight memory 2202 through the Direct Memory Access Controller (DMAC) 2205. Input data is also transferred to unified memory 2206 through the DMAC.

[0309] BIU stands for Bus Interface Unit, i.e., bus interface unit 2210 , which is used for interaction between the AXI bus, DMAC, and instruction fetch buffer (IFB) 2209 .

[0310] The bus interface unit 2210 (BIU) is used for the instruction fetch memory 2209 to obtain instructions from the external memory, and is also used for the storage unit access controller 2205 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0311] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 2206 or transfer weight data to the weight memory 2202 or transfer input data to the input memory 2201.

[0312] The vector calculation unit 2207 includes multiple operation processing units. When necessary, it further processes the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0313] In some implementations, the vector calculation unit 2207 can store the processed output vector to the unified memory 2206. For example, the vector calculation unit 2207 can apply a linear function and / or a nonlinear function to the output of the operation circuit 2203, such as linear interpolation of the feature plane extracted by the convolution layer, or accumulate a vector of values ​​to generate an activation value. In some implementations, the vector calculation unit 2207 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 2203, for example, for use in subsequent layers in a neural network.

[0314] An instruction fetch buffer 2209 connected to the controller 2204 is used to store instructions used by the controller 2204;

[0315] Unified memory 2206, input memory 2201, weight memory 2202, and instruction fetch memory 2209 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0316] in, Figures 2b to 15 The operations of each layer in the target model shown can be performed by the operation circuit 2203 or the vector calculation unit 2207.

[0317] The processor mentioned in any of the above places can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the program of the above-mentioned first aspect method.

[0318] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0319] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits or special circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0320] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0321] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a training device or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website, a computer, a training device or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

Claims

1. A method for processing point cloud data, characterized in that: The method comprises: Acquire point cloud data corresponding to the target environment, wherein the point cloud data includes initial information of a plurality of target points, the point cloud data is divided into a plurality of target cubes, each of the target cubes contains S target points, and S is an integer greater than or equal to 0; generating an initial feature of each target cube according to initial information of the target point in each target cube; updating the initial features of the plurality of target cubes according to the attention mechanism to obtain updated features of the plurality of target cubes; performing a feature processing operation on the updated features of the plurality of target cubes to obtain a prediction result corresponding to the point cloud data, the prediction result representing information of at least one object in the target environment; The updating of the initial features of the plurality of target cubes according to the attention mechanism includes: When a first cube satisfies a first preset condition, updating the features of each first cube in the multiple target cubes according to the attention mechanism, where the first cube is any one of the multiple target cubes, and the first preset condition is that the first cube is a non-empty cube, the initial features of the non-empty cube are not preset information, and the initial features of the empty cube are the preset information; In which, a hash table is used to record the non-empty cubes in the multiple target cubes, and the hash table corresponding to the multiple target cubes includes a keyword key and a value. The keyword corresponding to the target cube is obtained based on the index information of the target cube. The value corresponding to the target cube whose category is empty cube is a preset value. The value corresponding to the target cube whose category is non-empty cube is the order of the target cube whose category is non-empty cube among all non-empty cubes included in the multiple target cubes.

2. The method according to claim 1, characterized in that The updating of the initial features of the plurality of target cubes according to the attention mechanism further includes: When the first cube satisfies the first preset condition and the second preset condition, updating the features of each first cube in the plurality of target cubes according to the attention mechanism; The second preset condition is that the non-empty cube exists in the cube set, the cube set includes the first cube and at least one second cube corresponding to the first cube, and the at least one second cube is determined from the multiple target cubes according to the attention mechanism.

3. The method according to claim 2, characterized in that The at least one second cube includes all the target cubes within a preset range around the first cube, and the distance between the second cubes within the preset range around the first cube and the first cube is less than or equal to a preset distance threshold; and / or, The at least one second cube is obtained by sampling the multiple target cubes with the first cube as the sampling center.

4. The method according to claim 2, characterized in that The updating of features of each first cube in the plurality of target cubes according to the attention mechanism further includes: generating features of a plurality of third cubes according to the features of the second cube, wherein one third cube includes a plurality of the second cubes; According to the features of each of the third cubes, features of each of the first cubes in the multiple target cubes are updated based on a global attention mechanism, where the global attention mechanism is one of the attention mechanisms.

5. The method according to any one of claims 1 to 4, characterized in that The point cloud data corresponding to the target environment includes point cloud data corresponding to the environment surrounding the target vehicle, and performing a feature processing operation on the updated features of the target cube to obtain a prediction result corresponding to the point cloud data includes: A target detection operation is performed on the updated features of the target cube to obtain a prediction result corresponding to the point cloud data, where the prediction result is used to indicate a position of at least one object in the surrounding environment of the target vehicle.

6. The method according to any one of claims 1 to 4, characterized in that The performing a feature processing operation according to the updated features of the plurality of target cubes to obtain a prediction result corresponding to the point cloud data includes: performing a target detection operation on the updated features of the plurality of target cubes to obtain a prediction result corresponding to the point cloud data, wherein the information of the at least one object in the target environment includes position information of the at least one object in the target environment; or performing a face recognition operation on the updated features of the plurality of target cubes to obtain a prediction result corresponding to the point cloud data, wherein the information of the at least one object in the target environment includes category information of the at least one object in the target environment, and the category information of the at least one object in the target environment is used to indicate a success or failure of the face match; or Performing a posture recognition operation on the updated features of the multiple target cubes to obtain a prediction result corresponding to the point cloud data, wherein the information of at least one object in the target environment includes the physical features of at least one human body in the target environment and / or the gait features of at least one human body in the target environment when walking.

7. A neural network training method, characterized in that: The method comprises: Acquire point cloud data corresponding to the target environment, wherein the point cloud data includes initial information of a plurality of target points, the point cloud data is divided into a plurality of target cubes, and there are S target points in one target cube, where S is an integer greater than or equal to 0; According to the initial information of the target point in each target cube, an initial feature of each target cube is generated by the to-be-trained model, and the initial features of the multiple target cubes are updated according to the attention mechanism to obtain updated features of the multiple target cubes; performing a feature processing operation by the to-be-trained model according to the updated features of the plurality of target cubes to obtain a prediction result corresponding to the point cloud data; Training the to-be-trained model according to a target loss function, wherein the target loss function is used to indicate a similarity between the predicted result and an expected result corresponding to the point cloud data; The updating of the initial features of the plurality of target cubes according to the attention mechanism includes: When the first cube satisfies a first preset condition, updating the initial features of each first cube in the multiple target cubes according to the attention mechanism, the first cube is any one of the multiple target cubes, the first preset condition is that the first cube is a non-empty cube, the initial features of the non-empty cube are not preset information, and the initial features of the empty cube are the preset information; In which, a hash table is used to record the non-empty cubes in the multiple target cubes. The hash table corresponding to the multiple target cubes includes two parts: a keyword key and a value. The keyword corresponding to the target cube is obtained based on the index information of the target cube. The value corresponding to the target cube whose category is an empty cube is a preset value. The value corresponding to the target cube whose category is a non-empty cube is the order of the target cube whose category is a non-empty cube among all non-empty cubes included in the multiple target cubes.

8. The method according to claim 7, characterized in that The updating of the initial features of the plurality of target cubes according to the attention mechanism further includes: When the first cube satisfies the first preset condition and the second preset condition, updating the features of each first cube in the plurality of target cubes according to the attention mechanism; The second preset condition is that there is a non-empty cube in the cube set, the cube set includes the first cube and at least one second cube corresponding to the first cube, and the at least one second cube is determined from the multiple target cubes according to the attention mechanism.

9. A method for processing point cloud data, characterized in that: The method comprises: An operation indicating face recognition is detected; In response to the operation of instructing face recognition, outputting a face recognition result; Wherein, the face recognition result includes a successful match and a failed match, the face recognition result is obtained according to the updated features of multiple target cubes, the updated features of the target cube are obtained after updating the initial features of the target cube according to the attention mechanism, the initial features of the target cube are obtained based on the collected point cloud data corresponding to the target environment, the point cloud data includes initial information of multiple target points, the point cloud data is divided into multiple target cubes, there are S target points in one target cube, S is an integer greater than or equal to 0, the feature updating of the initial features of the target cube according to the attention mechanism includes: updating the initial features of a first cube in the multiple target cubes that meets a first preset condition according to the attention mechanism, the first cube is any one of the multiple target cubes, the first preset condition is that the first cube is a non-empty cube, the initial features of the non-empty cube are not preset information, and the initial features of the empty cube are the preset information; In which, a hash table is used to record the non-empty cubes in the multiple target cubes. The hash table corresponding to the multiple target cubes includes two parts: a keyword key and a value. The keyword corresponding to the target cube is obtained based on the index information of the target cube. The value corresponding to the target cube whose category is an empty cube is a preset value. The value corresponding to the target cube whose category is a non-empty cube is the order of the target cube whose category is a non-empty cube among all non-empty cubes included in the multiple target cubes.

10. The method according to claim 9, characterized in that The detecting of the operation indicating face recognition includes: An operation indicating face recognition is detected when any of the following operations is detected: unlocking the screen, making a payment, or opening encrypted data; or Detecting a door opening operation input by a user; or, A face matching operation is detected for user input.

11. A point cloud data processing device, characterized in that: The device comprises: an acquisition module, configured to acquire point cloud data corresponding to a target environment, wherein the point cloud data includes initial information of a plurality of target points, the point cloud data is divided into a plurality of target cubes, each of the target cubes contains S target points, and S is an integer greater than or equal to 0; A generating module, configured to generate initial features of each target cube according to initial information of target points in each target cube; An updating module, configured to update the initial features of the plurality of target cubes according to an attention mechanism to obtain updated features of the plurality of target cubes; a processing module, configured to perform a feature processing operation on the updated features of the plurality of target cubes to obtain a prediction result corresponding to the point cloud data, wherein the prediction result represents information of at least one object in the target environment; The updating module is specifically configured to update the features of each first cube in the multiple target cubes according to the attention mechanism when the first cube satisfies a first preset condition, the first cube being any one of the multiple target cubes, the first preset condition being a non-empty cube, the initial features of the non-empty cube not being the preset information, and the initial features of the empty cube being the preset information; In which, a hash table is used to record the non-empty cubes in the multiple target cubes. The hash table corresponding to the multiple target cubes includes two parts: a keyword key and a value. The keyword corresponding to the target cube is obtained based on the index information of the target cube. The value corresponding to the target cube whose category is an empty cube is a preset value. The value corresponding to the target cube whose category is a non-empty cube is the order of the target cube whose category is a non-empty cube among all non-empty cubes included in the multiple target cubes.

12. The device according to claim 11, characterized in that The updating module is further configured to update the features of each first cube in the plurality of target cubes according to the attention mechanism when the first cube satisfies the first preset condition and the second preset condition; The second preset condition is that there is a non-empty cube in the cube set, the cube set includes the first cube and at least one second cube corresponding to the first cube, and the at least one second cube is determined from the multiple target cubes according to the attention mechanism.

13. The device according to claim 12, characterized in that The at least one second cube includes all the target cubes within a preset range around the first cube, and the distance between the second cubes within the preset range around the first cube and the first cube is less than or equal to a preset distance threshold; and / or, The at least one second cube is obtained by sampling the multiple target cubes with the first cube as the sampling center.

14. The device according to claim 12, characterized in that The update module is further configured to: generating features of a plurality of third cubes according to the features of the second cube, wherein one third cube includes a plurality of the second cubes; According to the features of each of the third cubes, features of each of the first cubes in the multiple target cubes are updated based on a global attention mechanism, where the global attention mechanism is one of the attention mechanisms.

15. The device according to any one of claims 11 to 14, characterized in that The point cloud data corresponding to the target environment includes point cloud data corresponding to the surrounding environment of the target vehicle; The processing module is specifically used to perform a target detection operation on the updated features of the target cube to obtain a prediction result corresponding to the point cloud data, and the prediction result is used to indicate the position of at least one object in the environment surrounding the target vehicle.

16. The device according to any one of claims 11 to 14, characterized in that The processing module is specifically used to: performing a target detection operation on the updated features of the plurality of target cubes to obtain a prediction result corresponding to the point cloud data, wherein the information of the at least one object in the target environment includes position information of the at least one object in the target environment; or, performing a face recognition operation on the updated features of the plurality of target cubes to obtain a prediction result corresponding to the point cloud data, wherein the information of the at least one object in the target environment includes category information of the at least one object in the target environment, and the category information of the at least one object in the target environment is used to indicate a success or failure of the face match; or Performing a posture recognition operation on the updated features of the multiple target cubes to obtain a prediction result corresponding to the point cloud data, wherein the information of at least one object in the target environment includes the physical features of at least one human body in the target environment and / or the gait features of at least one human body in the target environment when walking.

17. A neural network training device, characterized in that: The device comprises: an acquisition module, configured to acquire point cloud data corresponding to a target environment, wherein the point cloud data includes initial information of a plurality of target points, the point cloud data is divided into a plurality of target cubes, and there are S target points in one target cube, where S is an integer greater than or equal to 0; A generating module, configured to generate initial features of each target cube using a to-be-trained model according to initial information of target points within each target cube; An updating module, configured to update the initial features of the plurality of target cubes according to an attention mechanism to obtain updated features of the plurality of target cubes; a processing module, configured to perform a feature processing operation using the to-be-trained model based on the updated features of the plurality of target cubes to obtain a prediction result corresponding to the point cloud data; a training module, configured to train the model to be trained according to a target loss function, wherein the target loss function is used to indicate a similarity between the predicted result and an expected result corresponding to the point cloud data; The updating module is specifically configured to update the features of each first cube in the multiple target cubes according to the attention mechanism when the first cube satisfies a first preset condition, the first cube being any one of the multiple target cubes, the first preset condition being a non-empty cube, the initial features of the non-empty cube not being the preset information, and the initial features of the empty cube being the preset information; In which, a hash table is used to record the non-empty cubes in the multiple target cubes. The hash table corresponding to the multiple target cubes includes two parts: a keyword key and a value. The keyword corresponding to the target cube is obtained based on the index information of the target cube. The value corresponding to the target cube whose category is an empty cube is a preset value. The value corresponding to the target cube whose category is a non-empty cube is the order of the target cube whose category is a non-empty cube among all non-empty cubes included in the multiple target cubes.

18. A point cloud data processing device, characterized in that: The device comprises: A detection module, configured to detect an operation indicating face recognition; An output module, configured to output a face recognition result in response to the operation of instructing face recognition; Wherein, the face recognition result includes a successful match and a failed match, the face recognition result is obtained according to the updated features of multiple target cubes, the updated features of the target cube are obtained after updating the initial features of the target cube according to the attention mechanism, the initial features of the target cube are obtained based on the collected point cloud data corresponding to the target environment, the point cloud data includes initial information of multiple target points, the point cloud data is divided into multiple target cubes, there are S target points in one target cube, S is an integer greater than or equal to 0, the feature updating of the initial features of the target cube according to the attention mechanism includes: updating the initial features of a first cube in the multiple target cubes that meets a first preset condition according to the attention mechanism, the first cube is any one of the multiple target cubes, the first preset condition is that the first cube is a non-empty cube, the initial features of the non-empty cube are not preset information, and the initial features of the empty cube are the preset information; In which, a hash table is used to record the non-empty cubes in the multiple target cubes. The hash table corresponding to the multiple target cubes includes two parts: a keyword key and a value. The keyword corresponding to the target cube is obtained based on the index information of the target cube. The value corresponding to the target cube whose category is an empty cube is a preset value. The value corresponding to the target cube whose category is a non-empty cube is the order of the target cube whose category is a non-empty cube among all non-empty cubes included in the multiple target cubes.

19. A computer program product, characterized in that When the computer program is run on a computer, the computer is caused to perform the method according to any one of claims 1 to 10.

20. A computer-readable storage medium, characterized in that The invention comprises a program, which, when being run on a computer, causes the computer to execute the method according to any one of claims 1 to 10.

21. An execution device, characterized in that: comprising a processor and a memory, the processor being coupled to the memory, The memory is used to store programs; The processor is configured to execute the program in the memory, so that the execution device executes the method according to any one of claims 1 to 6; or, the execution device executes the method according to claim 9 or 10.

22. A training device, characterized in that comprising a processor and a memory, the processor being coupled to the memory, The memory is used to store programs; The processor is configured to execute the program in the memory so that the training device performs the method according to claim 7 or 8.

Images