Processing device, processing method and mobile robot

Abstract

The invention provides a processing device, a processing method and a mobile robot, is applied to the technical field of mobile robots, and aims to reduce the occupancy rate and power consumption of a central processing unit while improving the processing efficiency of depth image-to-point cloud data. Wherein the processing device comprises a central processing unit, an artificial intelligence processor and a memory. The central processing unit can perform matrix operation according to an external parameter matrix and an internal parameter matrix of the camera to obtain a first operation result. The artificial intelligence processor can obtain the depth image and the pixel coordinates corresponding to the depth image, and generates point cloud data according to the depth image, the pixel coordinates and the first operation result. The memory can transfer the first operation result and the point cloud data between the central processor and the artificial intelligence processor. On the basis, the occupancy rate and power consumption of the central processing unit can be greatly reduced, and meanwhile, the advantage of high processing speed of the artificial intelligence processor can be fully utilized to improve the efficiency of converting the depth image into the point cloud data.

Description

Technical Field

[0001] This application relates to the field of mobile robot technology, and in particular to a processing device, processing method and mobile robot. Background Technology

[0002] Depth map to point cloud data is the process of calculating the world coordinates of each pixel in a depth map based on its depth information. This allows us to obtain the coordinates of each pixel in the depth map within three-dimensional space, thus determining the position of an object in its world coordinate system. In scenarios involving ground mobile robots (such as autonomous vehicles, ground inspection robots, ground cleaning robots, ground logistics robots, and ground agricultural robots), aerial mobile robots (such as insect-inspired flying mobile robots, flying mobile robots, quadcopters, and micro-aircraft), and water mobile robots, depth map to point cloud data can be used for obstacle avoidance, path planning, and 3D modeling. It is significant for rapidly identifying object positions, controlling movement speed, acceleration, and distance traveled. Currently, depth map to point cloud data is primarily implemented by executing the algorithm on a central processing unit (CPU). The larger the depth map, the greater the computational load on the CPU, leading to lower processing efficiency, higher CPU utilization, and consequently reduced CPU efficiency for other tasks. Therefore, a solution to address these issues is urgently needed. Summary of the Invention

[0003] This application provides a processing device, processing method, and mobile robot to improve the processing efficiency of depth image to point cloud data while reducing the utilization and power consumption of the central processing unit.

[0004] To achieve the above objectives, the embodiments of this application provide the following technical solutions.

[0005] In a first aspect, a processing apparatus is provided, comprising a central processing unit (CPU), an artificial intelligence (AI) processor, and a memory. The CPU performs matrix operations based on the extrinsic and intrinsic parameters of a camera to obtain a first calculation result. The extrinsic parameters are transformation matrices used to convert pixels in the depth image from the world coordinate system to the camera coordinate system, and the intrinsic parameters are transformation matrices used to convert pixels in the depth image from the camera coordinate system to the pixel coordinate system. The AI ​​processor acquires the depth image and the corresponding pixel coordinates, and generates point cloud data based on the depth image, the pixel coordinates, and the first calculation result. The memory is coupled to the CPU and the AI ​​processor, and can transfer the first calculation result and the point cloud data between the CPU and the AI ​​processor.

[0006] By employing the above method, compared to existing solutions, the central processing unit (CPU) only needs to perform matrix operations based on the camera's extrinsic and intrinsic parameter matrices, while the large-scale pixel-by-pixel calculations in the depth image are assigned to the artificial intelligence (AI) processor. Because the matrix operations on the extrinsic and intrinsic parameter matrices are relatively small, the CPU's occupancy and power consumption can be significantly reduced. Simultaneously, the high processing speed of the AI ​​processor can be fully utilized to improve the efficiency of converting depth images into point cloud data.

[0007] In one possible implementation, the first operation result includes a first matrix and a second matrix, where the first matrix is ​​a 3x3 matrix. The AI ​​processor can first split the first matrix into multiple sub-matrices. Then, it performs multiplication and addition operations based on the depth image, pixel coordinates, the second matrix, and the multiple sub-matrices to generate point cloud data. The AI ​​processor is specifically designed to execute neural network algorithms; however, depth-to-point cloud computing is a non-neural network algorithm and cannot be directly executed on the AI ​​processor. Through this method, the AI ​​processor can split the matrix operation result from the central processing unit into multiple sub-matrices before performing multiplication and addition operations to generate point cloud data. Based on this, computational operations can be broken down into neural network operators supported by the AI ​​processor, enabling the depth image-to-point cloud algorithm to be executed on the AI ​​processor, thus improving the portability of the accelerated algorithm.

[0008] In one possible implementation, the central processing unit (CPU) can also generate the pixel coordinates corresponding to each pixel in the depth image based on the depth image. The memory can also transfer the pixel coordinates between the CPU and the AI ​​processor. Calculating the pixel coordinates of each pixel in the depth image only requires calculation based on the width and height of the depth image, resulting in relatively low computational complexity. Therefore, having the CPU calculate the pixel coordinates of each pixel in the depth image ensures computational efficiency while reducing the complexity of the neural network in the AI ​​processor.

[0009] In one possible implementation, the AI ​​processor can first generate a horizontal coordinate matrix and a vertical coordinate matrix based on the aforementioned pixel coordinates. Then, it performs a first multiplication-addition operation on the horizontal coordinate matrix, the vertical coordinate matrix, and multiple sub-matrices to obtain a second operation result. Finally, it performs a second multiplication-addition operation on the second operation result, the second matrix, and the depth image to obtain point cloud data. Through this method, a depth image to point cloud data conversion algorithm can be implemented on the AI ​​processor using multiplication-addition operations directly between multiple matrices, thereby improving the computational efficiency of depth image to point cloud data conversion.

[0010] In one possible implementation, the aforementioned processing device is a chip, in which an artificial intelligence processor, a central processing unit, and a memory are integrated.

[0011] In one possible implementation, the depth image is stored in memory as image blocks and transferred to the artificial intelligence processor as image blocks.

[0012] Secondly, a processing method is provided. During execution, this method first uses a central processing unit (CPU) to perform matrix operations based on the camera's extrinsic and intrinsic parameter matrices to obtain a first calculation result. The extrinsic parameter matrix is ​​the transformation matrix for pixels in the depth image from the world coordinate system to the camera coordinate system, and the intrinsic parameter matrix is ​​the transformation matrix for pixels in the depth image from the camera coordinate system to the pixel coordinate system. Then, the first calculation result is transferred between the CPU and the AI ​​processor via memory. Next, the AI ​​processor acquires the depth image and its corresponding pixel coordinates, and generates point cloud data based on the depth image, pixel coordinates, and the first calculation result. Finally, the point cloud data is transferred between the AI ​​processor and the CPU via memory.

[0013] In one possible implementation, when the AI ​​processor acquires a depth image and the corresponding pixel coordinates, and generates point cloud data based on the depth image, pixel coordinates, and the first calculation result, it can first split the first matrix into multiple sub-matrices. Then, the AI ​​processor performs multiplication and addition operations based on the depth image, pixel coordinates, the second matrix, and the multiple sub-matrices to generate point cloud data.

[0014] In one possible implementation, the method further includes: generating pixel coordinates corresponding to the depth image using a central processing unit (CPU); and transferring the pixel coordinates between the CPU and the AI ​​processor using memory.

[0015] In one possible implementation, when the AI ​​processor generates point cloud data based on the depth image, pixel coordinates, and the first calculation result, it can first generate a horizontal coordinate matrix and a vertical coordinate matrix based on the pixel coordinates. Then, it performs a first multiplication-addition operation based on the horizontal coordinate matrix, the vertical coordinate matrix, and multiple sub-matrices to obtain a second calculation result. Finally, it performs a second multiplication-addition operation based on the second calculation result, the second matrix, and the depth image to obtain the point cloud data.

[0016] In one possible implementation, the above method is applied to a processing device, which is a chip, in which an artificial intelligence processor, a central processing unit, and a memory are integrated.

[0017] In one possible implementation, the depth image is stored in memory as image blocks and transferred to the artificial intelligence processor as image blocks.

[0018] Thirdly, a mobile robot is provided, comprising a camera and a processing device connected to the camera. The processing device includes a central processing unit (CPU), an artificial intelligence (AI) processor, and a memory. The CPU can perform matrix operations based on the camera's extrinsic and intrinsic parameter matrices to obtain a first calculation result. The extrinsic parameter matrix is ​​a transformation matrix for converting pixels in the depth image from the world coordinate system to the camera coordinate system, and the intrinsic parameter matrix is ​​a transformation matrix for converting pixels in the depth image from the camera coordinate system to the pixel coordinate system. The AI ​​processor can acquire the depth image and the corresponding pixel coordinates, and generate point cloud data based on the depth image, pixel coordinates, and the first calculation result. The memory is coupled to the CPU and the AI ​​processor, and can transfer the first calculation result and the point cloud data between the CPU and the AI ​​processor.

[0019] In one possible implementation, the mobile robot further includes a controller connected to a processing device, and a gyroscope and an accelerometer connected to the controller. The controller can acquire the angular velocity detected by the gyroscope and the acceleration detected by the accelerometer. Then, based on the angular velocity, the acceleration, and the point cloud data, the controller controls the mobile robot's direction and speed of movement.

[0020] The technical effects brought about by the second to third aspects and possible implementations described above can be found in the description of the technical effects brought about by the first aspect and possible implementations described above, and will not be repeated here. Attached Figure Description

[0021] Figure 1 This is a schematic diagram illustrating the principle of converting depth images to point cloud data.

[0022] Figure 2 A schematic diagram of a process for converting depth images into point cloud data;

[0023] Figure 3 A schematic diagram of another process for converting depth images into point cloud data;

[0024] Figure 4 This is a schematic diagram of the data flow of a processing device provided in an embodiment of this application;

[0025] Figure 5 This application provides a schematic diagram of a process for converting depth images into point cloud data.

[0026] Figure 6 This is a schematic diagram of the data flow of another processing device provided in an embodiment of this application;

[0027] Figure 7This application provides a schematic diagram of a network model structure for an artificial intelligence processor.

[0028] Figure 8 This is a schematic diagram of the structure of a mobile robot provided in an embodiment of this application. Detailed Implementation

[0029] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0030] To facilitate a clear description of the technical solutions in the embodiments of this application, the terms "first" and "second" are used to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, nor do they necessarily imply differences. Furthermore, in the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as superior or more advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present related concepts in a concrete manner for ease of understanding. In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more. For example, multiple processors means two or more processors.

[0031] In describing some embodiments, the terms "coupled" and "connected," and their derivative expressions, may be used. For example, the term "connected" may be used in describing some embodiments to indicate that two or more components have direct physical contact or point contact with each other. Similarly, the term "coupled" may be used in describing some embodiments to indicate that two or more components have direct physical contact or electrical contact, or it may refer to two or more components that do not have direct contact with each other but still cooperate or interact with each other. The embodiments disclosed herein are not necessarily limited to the content of this document.

[0032] In addition, for ease of understanding, the technical terms involved in the embodiments of this application will be introduced below.

[0033] Depth map: A depth map is an image format used to represent the depth information of a 3D scene. It encodes the depth value of each pixel in the 3D scene into a grayscale value and presents it as a 2D image. In a depth map, areas with higher grayscale values ​​represent objects closer to the camera, and areas with lower grayscale values ​​represent objects farther away from the camera.

[0034] Point cloud data is a data format used to represent three-dimensional structures. It represents objects or scenes in three-dimensional space as points. Each point has three coordinates: x, y, and z, representing its position in three-dimensional space. Point cloud data may also include other information such as color, intensity, or normals. Unlike traditional 3D models, it does not have continuous surfaces but is composed of discrete points. Point cloud data can be used to represent physical objects with three-dimensional structures and is a commonly used data representation method in fields such as 3D reconstruction and computer graphics.

[0035] Intrinsic parameters: Intrinsic parameters describe the internal properties of a camera, including focal length, principal point (optical center) coordinates, distortion coefficients, etc. Intrinsic parameters are usually determined during camera calibration because they are typically fixed for a specific camera model and do not change over time. Once the camera's intrinsic parameters are determined, they generally remain unchanged during the camera's use. The camera's intrinsic parameters are usually represented by an intrinsic parameter matrix, K, which is generally expressed as:

[0036]

[0037] Among them, f x f represents the horizontal focal length of the camera; y Indicates the camera's vertical focal length; c x This represents the x-coordinate of the camera's principal point, that is, the x-coordinate of the origin of the image coordinate system in the pixel coordinate system; c y This represents the ordinate of the camera's principal point, which is the ordinate of the origin of the image coordinate system in the pixel coordinate system.

[0038] Extrinsic parameters: Extrinsic parameters describe the position and orientation of the camera in the world coordinate system. These parameters may change depending on the camera position or the moment of capture. For example, in stereo vision, if there are two cameras, their relative position and orientation will change with each camera movement, resulting in changes in the extrinsic parameters. Extrinsic parameters are typically represented by an extrinsic matrix, which includes a rotation matrix R and a translation matrix T. The rotation matrix R represents the relative rotation between the world coordinate system and the camera coordinate system; this matrix is ​​typically a 3x3 matrix. The translation matrix T represents the relative displacement between the world coordinate system and the camera coordinate system; this matrix is ​​typically a 3x1 matrix. The rotation matrix R can be represented as:

[0039]

[0040] Where, r00 ~r 22 This represents the elements in the rotation matrix R.

[0041] The translation matrix can be represented as:

[0042]

[0043] Where t0 to t2 represent the elements in the rotation matrix T.

[0044] The extrinsic parameter matrix can be represented as:

[0045]

[0046] Sometimes, the [0 0 0 1] in the last line can be ignored.

[0047] The present application will now be described in detail with reference to the accompanying drawings and embodiments:

[0048] With the continuous development of artificial intelligence (AI) technology, various types of mobile robots are becoming increasingly intelligent. Point cloud data possesses high-precision, high-resolution, and high-dimensional geometric information, which can intuitively represent the shape, surface, and texture of objects in space. Therefore, many mobile robots use point cloud data as the foundation for functions such as obstacle avoidance, path planning, and 3D modeling. Based on this, the efficiency of point cloud data acquisition is crucial for quickly identifying object positions, controlling movement speed, acceleration, and movement distance. Currently, point cloud data is mainly acquired through two methods: LiDAR acquisition and depth image acquisition. In scenarios involving ground mobile robots (such as autonomous vehicles, ground inspection robots, ground cleaning robots, ground logistics robots, and ground agricultural robots) and aerial mobile robots (such as insect-inspired flying mobile robots, flying mobile robots, quadcopters, and micro-aircraft), point cloud data is often acquired by converting depth images into point cloud data.

[0049] For example, the transformation process of world coordinates for each pixel in a depth image is as follows: Figure 1 As shown. In Figure 1 The image on the left shows the depth of a cube-shaped object. Different colors represent different distances. Figure 1 The image on the right shows the coordinates of the top-left pixel of the cube in the world coordinate system, calculated based on the depth information from the depth image. When converting the depth image to point cloud data, the pixel coordinates (u, v) and world coordinates (x, y) of each pixel are compared. w ,y w , z w The conversion relationship between them is shown in equation (1) below:

[0050]

[0051] Based on the above transformation relationship, the world coordinates of each pixel in the depth image can be calculated as shown in equation (2) below:

[0052]

[0053] In the above implementation process, the rotation matrix can be obtained in the following way:

[0054] 1) Set some specific points or features in a known world coordinate system, then use a camera to acquire a depth image and detect the pixel coordinates of these points in the depth image.

[0055] 2) Select a set of calibration points from the known world coordinate points and their corresponding points in the depth image. Typically, at least three non-collinear points are required to calculate the rotation matrix.

[0056] 3) For each selected calibration point, convert the pixel coordinates (u, v) in the depth image to normalized planar coordinates.

[0057] 4) The rotation matrix can be calculated using the world coordinates of the selected calibration point and the normalized plane coordinates. A common method is to use SVD (Singular Value Decomposition) to calculate the rotation matrix.

[0058] The translation matrix can be obtained in the following way:

[0059] 1) Set some specific points or features in a known world coordinate system, then use a camera to acquire a depth image and detect the pixel coordinates of these points in the depth image.

[0060] 2) Select a set of calibration points from the known world coordinate points and their corresponding points in the depth image. Typically, at least three non-collinear points are needed to calculate the translation vector.

[0061] 3) For each selected calibration point, convert the pixel coordinates (u, v) in the depth image to normalized planar coordinates. This can be achieved using the camera's intrinsic parameters.

[0062] 4) Using the world coordinates of the selected calibration point and the normalized plane coordinates, solve for the translation matrix using linear or nonlinear least squares methods.

[0063] Currently, to convert depth images into point cloud data using the methods described above, the depth image to point cloud data conversion algorithm is typically implemented using a CPU or digital signal processor (DSP). For example... Figure 2As shown, when using the CPU to convert a depth image into point cloud data, the image can be scaled down after acquiring the original depth image. Then, the CPU processes the scaled-down image to obtain preliminary point cloud data. Next, this preliminary point cloud data is mapped back to the depth image to obtain the world coordinates of each pixel in the depth image, resulting in the final point cloud data. However, this approach has high CPU usage, requiring powerful CPU capabilities, and its processing efficiency is relatively low, making it difficult to meet the demands of high frame rate scenarios. Furthermore, scaling also affects the accuracy of the point cloud data. High CPU usage can also impact other business processes on chips with insufficient CPU computing power.

[0064] like Figure 3 As shown, when using a DSP to convert a depth image into point cloud data, the depth image can be segmented into image blocks after acquisition. Then, the DSP processes these image blocks to obtain the corresponding point cloud data. However, this approach requires a dedicated parallel processor, which is expensive, and many chips lack this parallel processing core, or the parallel processor models differ (e.g., different DSP models use different acceleration instructions), resulting in poor versatility of the acceleration solution.

[0065] To solve the above problems, such as Figure 4 As shown in the illustration, this application provides a processing device 400, which includes a central processing unit 410, an artificial intelligence processor 420 (or AI processor), and a memory 430. The central processing unit 410 can perform matrix operations based on the extrinsic and intrinsic parameter matrices of a camera 500 to obtain a first calculation result. The extrinsic parameter matrix is ​​a transformation matrix for converting pixels in the depth image from the world coordinate system to the camera coordinate system, and the intrinsic parameter matrix is ​​a transformation matrix for converting pixels in the depth image from the camera coordinate system to the pixel coordinate system. The artificial intelligence processor 420 can acquire the depth image and the corresponding pixel coordinates, and generate point cloud data based on the depth image, pixel coordinates, and the first calculation result. The memory 430 is coupled to the central processing unit 410 and the artificial intelligence processor 420, and is used to transfer the first calculation result and the point cloud data between the central processing unit 410 and the artificial intelligence processor 420.

[0066] In this way, compared with existing solutions, the central processing unit 410 only needs to perform matrix operations based on the camera's extrinsic and intrinsic parameter matrices, while the large-scale pixel-by-pixel calculations in the depth image are assigned to the artificial intelligence processor 420. Because the matrix operations of the extrinsic and intrinsic parameter matrices are very small, the utilization rate and power consumption of the central processing unit 410 can be greatly reduced. At the same time, the high processing speed of the artificial intelligence processor 420 can be fully utilized to improve the efficiency of converting depth images into point cloud data.

[0067] In the above embodiments, the artificial intelligence processor 420 includes, but is not limited to, a neural processing unit (NPU), a tensor processing unit (TPU), or a neural processing engine (NPE). These multiple processors can be integrated into one or more chips, which can be considered a chipset. When multiple processors are integrated into the same chip, the chip is also called a system on a chip (SOC). In addition to the multiple processors, the processing device 400 also includes one or more other necessary components, such as a memory 430.

[0068] In one possible implementation, the aforementioned memory 430 may reside within the same system-on-a-chip as the artificial intelligence processor 420 and the central processing unit 410 within the processing device 400; that is, the memory 430 is integrated into the aforementioned... Figure 4 The processing device 400 shown uses on-chip memory. In this case, the memory 430 can be random access memory (RAM). This RAM can be volatile memory, such as SRAM, DRAM, DDR, synchronous dynamic random-access memory (SDRAM), or double data rate SDRAM. It can also be non-volatile random access memory (NVRAM). The on-chip memory can store an executable program for a depth image-to-point cloud data processing model that runs in the artificial intelligence processor. The artificial intelligence processor 420 loads this executable program to run the processing model.

[0069] In another possible implementation, the aforementioned memory 430 can also be off-chip memory. This off-chip memory can be RAM. The random access memory can be volatile memory, such as SRAM, DRAM, DDR, SDRAM, Double Data Rate SDRAM, etc. The random access memory can also be NVRAM. Furthermore, the aforementioned off-chip memory can store an executable program for a depth image-to-point cloud data processing model that runs in the artificial intelligence processor 420, which loads the executable program to run the processing model.

[0070] In one implementation scheme, such as Figure 5 As shown, this application embodiment also provides a processing method applied to the above-described processing apparatus 400. The execution process of this processing method is as follows.

[0071] S501: The central processing unit performs matrix operations based on the camera's extrinsic and intrinsic parameter matrices to obtain the first operation result.

[0072] As mentioned above, the intrinsic parameters of camera 500 include its focal length, principal point (optical center) coordinates, and distortion coefficients. These intrinsic parameters are typically determined during camera 500 calibration and are usually fixed for a specific camera model, remaining unchanged over time. Once determined, the intrinsic parameters of camera 500 generally remain constant during its use. Therefore, the intrinsic parameter matrix of camera 500 can be determined based on its internal parameters such as focal length, principal point (optical center) coordinates, and distortion coefficients, and the elements within this matrix are fixed values ​​that can be set during camera 500 calibration. The extrinsic parameters of camera 500 are the parameters determining its position and orientation in the world coordinate system. These extrinsic parameters may change depending on the camera's position or the time of shooting. The extrinsic parameters of camera 500 are typically represented by an extrinsic parameter matrix, which includes rotation and translation matrices. Based on this, when the central processing unit 410 performs matrix operations based on the external and internal parameter matrices of the camera 500, it only needs to execute a portion of the operations in equation (2) above, that is, to execute matrix operation R. -1 K -1 and R -1 T. By performing the above matrix operations, two matrices can be obtained. Therefore, the result of the first operation can include a first matrix and a second matrix. The first matrix is ​​a multi-row, multi-column matrix, and the second matrix is ​​a single-row, multi-column matrix. For example, since the rotation matrix R is usually a 3x3 matrix, the translation matrix T is usually a 3x1 matrix, and the intrinsic parameter matrix K is also a 3x3 matrix, the first matrix is ​​a 3x3 matrix, and the second matrix is ​​a 3x1 matrix. The first matrix can be represented as:

[0073]

[0074] Among them, a 00 ~a 22 These are the elements in the first matrix.

[0075] The second matrix mentioned above can be represented as:

[0076]

[0077] Among them, b 00 ~b20 These are the elements in the first matrix.

[0078] S502: The memory transfers the first operation result between the central processing unit and the artificial intelligence processor.

[0079] Since the first operation result includes a first matrix and a second matrix, the memory 430 can transfer the first operation result from the central processing unit 410 to the artificial intelligence processor 420 in the form of a matrix array.

[0080] S503: The artificial intelligence processor acquires a depth image and the corresponding pixel coordinates, and generates point cloud data based on the depth image, pixel coordinates, and the first calculation result.

[0081] In one possible implementation, such as Figure 6 As shown, the central processing unit 410 can also generate pixel coordinates corresponding to each pixel in the depth image based on the depth image. Then, the pixel coordinates are transferred from the central processing unit to the artificial intelligence processor 420 through the memory 430. Specifically, the central processing unit 410 can generate pixel coordinates (u, v) corresponding to each pixel in the depth image based on the width dx and height dy of the depth image. The value range of u is [0, dx-1], and the value range of v is [0, dy-1].

[0082] The central processing unit 410 can connect to a camera and control the camera 500 to acquire a depth image. This depth image can be stored in a designated area of ​​the memory 430. After storing the pixel coordinates and the first calculation result in the designated area of ​​the memory 430, the central processing unit 410 can send an instruction signal to the artificial intelligence processor 420. This instruction signal instructs the artificial intelligence processor 420 to acquire the depth image, the first calculation result, and the pixel coordinates from the designated area. In this implementation process, the depth image can be stored in the memory 430 in the form of image blocks and transferred to the artificial intelligence processor 420 in the form of image blocks.

[0083] In one possible implementation, the artificial intelligence processor 420 can also generate the pixel coordinates corresponding to each pixel in the depth image based on the width and height of the depth image. Both methods described above can be adjusted according to actual needs, and this application embodiment does not impose specific limitations on them.

[0084] In one possible implementation, such as Figure 7As shown, considering that the AI ​​processor 420 typically executes neural network algorithms, while the depth image to point cloud algorithm is a non-neural network algorithm, in order to enable its execution on the AI ​​processor 420, when the AI ​​processor 420 generates point cloud data based on the depth image, pixel coordinates, and the first calculation result, it can first split the aforementioned first matrix into multiple sub-matrices. Then, it performs multiplication and addition operations based on the depth image, pixel coordinates, the second matrix, and the aforementioned multiple sub-matrices to generate point cloud data.

[0085] In one implementation, the AI ​​processor 420 can generate a horizontal coordinate matrix and a vertical coordinate matrix based on the aforementioned pixel coordinates. Then, a first multiplication-addition operation is performed based on the horizontal coordinate matrix, the vertical coordinate matrix, and multiple sub-matrices to obtain a second operation result. Finally, a second multiplication-addition operation is performed based on the second operation result, the second matrix, and the depth image to obtain point cloud data. For example, as shown... Figure 7 As shown, the depth image is denoted as D1, and the second matrix is ​​denoted as B1. The artificial intelligence processor 420 can generate an abscissa matrix U1 and a ordinate matrix V1 based on the pixel coordinates of each pixel in the depth image. Simultaneously, it can split each column of the first matrix into a submatrix, obtaining submatrices A1, A2, and A3. Then, a first multiplication-addition operation is performed on the abscissa matrix U1, the ordinate matrix V1, and submatrices A1, A2, and A3. This first multiplication-addition operation includes: firstly, performing a neural network multiplication operation on the abscissa matrix U1 and submatrix A1 to obtain the result C1 = A1 * U1; and secondly, performing a neural network multiplication operation on the ordinate matrix V1 and submatrix A2 (i.e.,...). Figure 7 The result of the operation (MUL in the neural network) is obtained as C2 = A2 * V1. Next, the addition operation (i.e., addition in the neural network) is performed on C1 and C2. Figure 7 The process begins by using ADD to obtain C3, which is defined as C3 = A1*U1 + A2*V1. Then, an addition operation is performed on submatrices A3 and C3 to obtain the second result P. Finally, a second multiplication-addition operation is performed on the second result P, the depth image D1, and the second matrix B1 to obtain the point cloud data. This second multiplication-addition operation includes: first, multiplying the second result P and the depth image D1 to obtain the result C4; then, adding the second matrix B1 and the result C4 to obtain the corresponding point cloud data.

[0086] In the above manner, an executable network model for the AI ​​processor 420 can be constructed based on the split matrix and the multiply-add operators in the neural network, thereby reducing the utilization and power consumption of the central processing unit 410. Simultaneously, the high processing speed of the AI ​​processor 420 can be fully utilized to improve the efficiency of converting depth images into point cloud data. Of course, the above-described submatrix splitting method for the first matrix is ​​merely an exemplary implementation method provided in this application embodiment. In specific implementations, other splitting methods can also be used, and this application embodiment does not impose specific limitations on this.

[0087] S504: The memory transfers point cloud data between the AI ​​processor and the central processing unit.

[0088] The artificial intelligence processor 420 can store point cloud data in a designated area of ​​the memory. Then, the central processing unit 410 controls the mobile robot based on the point cloud data according to the actual application scenario.

[0089] In some implementation schemes, such as Figure 8As shown in the illustration, this application also provides a mobile robot 800. The aforementioned processing device 400 may be a module, chip, chip system, circuit board, or component integrated within the mobile robot 800. The mobile robot 800 may be a ground mobile robot, an aerial mobile robot, or a water mobile robot, etc. Specifically, the ground mobile robot may be an unmanned vehicle, a ground inspection robot, a ground cleaning robot, a ground logistics robot, or a ground agricultural robot, etc. The aerial robot may be an insect-inspired flying mobile robot, a flying mobile robot, a quadcopter, or a micro-aircraft, etc. The water mobile robot may be a water rescue robot, a water surface inspection robot, or an underwater robot, etc. The mobile robot 800 may be equipped with a camera 500, which may be a stereo vision camera (e.g., a binocular vision camera), a depth camera, a time-of-flight camera, etc., for acquiring depth images. The stereo vision camera uses two or more cameras to simultaneously capture the same scene from different angles. By analyzing the parallax between these images (i.e., the positional difference of the same point in the image under different viewpoints), the depth information of the object can be calculated. A depth camera is a sensor that integrates an infrared emitter and an infrared camera. It obtains depth information by emitting infrared light or structured light and measuring its reflection or deformation from an object. A Time-of-Flight (TOF) camera calculates distance by measuring the time required for a light pulse to travel from the camera to the object's reflection back to the camera. A TOF camera emits modulated infrared light pulses and determines the object's depth information by measuring the time difference of the light. The camera 500 can be connected to the processing device 400 via a depth map processor (not shown). This depth map processor processes the detection data acquired by the camera 500 to obtain a depth image. After storing the depth image in the memory 430, the depth map processor can send an instruction signal to the central processing unit 410 to instruct the central processing unit 410 to perform subsequent processing. Based on this, the computational load of the central processing unit 410 can be reduced, lowering its occupancy and power consumption.

[0090] In one embodiment, the mobile robot 800 may further include a controller 810 connected to the processing device 400, and an accelerometer 820 and a gyroscope 830 connected to the controller 810. The controller 810 can acquire the angular velocity detected by the gyroscope 830 and the acceleration detected by the accelerometer 820. Then, the controller 810 can control the movement direction and speed of the mobile robot 800 based on the acquired angular velocity, acceleration, and point cloud data. The acceleration detected by the accelerometer, the angular velocity detected by the gyroscope, and the depth image acquired by the camera can be calculated by the visual-inertial odometry (VIO) module in the CPU to determine the position, attitude, and speed of the mobile robot 800. Then, the CPU can send an execution signal to the controller 810 through a configured control module to instruct the controller 810 to control the movement direction and speed of the mobile robot 800.

[0091] In the above implementation process, when the computing power of the central processing unit is sufficient to meet the requirements, accelerometers and gyroscopes can also be directly connected to the central processing unit, thereby reducing the use of controller 810 and reducing cost and power consumption.

[0092] Understandably, the method steps in this embodiment can be implemented by a processor executing software instructions. These software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in the terminal device. Of course, the processor and storage medium can also exist as discrete components in the terminal device.

[0093] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. A computer program product includes one or more computer programs or instructions. When a computer program or instruction is loaded and executed on a computer, the processes or functions of the embodiments of this application are performed entirely or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, or other programmable device. The computer program or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, a computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; it can also be an optical medium, such as a digital video disc (DVD); or it can be a semiconductor medium, such as a solid-state drive (SSD).

[0094] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A processing apparatus, characterized in that, include: The central processing unit is used to perform matrix operations based on the camera's extrinsic and intrinsic parameter matrices to obtain the first calculation result; The extrinsic parameter matrix is ​​the transformation matrix for pixels in the depth image from the world coordinate system to the camera coordinate system; the intrinsic parameter matrix is ​​the transformation matrix for pixels in the depth image from the camera coordinate system to the pixel coordinate system. An artificial intelligence processor is used to acquire a depth image and the pixel coordinates corresponding to the depth image, and generate point cloud data based on the depth image, the pixel coordinates and the first calculation result; A memory, coupled to the central processing unit and the artificial intelligence processor, is used to transfer the first calculation result and the point cloud data between the central processing unit and the artificial intelligence processor.

2. The processing apparatus according to claim 1, characterized in that, The first calculation result includes a first matrix and a second matrix, wherein the first matrix is ​​a 3x3 matrix; the artificial intelligence processor is used for: The first matrix is ​​split into multiple sub-matrices; The point cloud data is generated by performing multiplication and addition operations on the depth image, the pixel coordinates, the second matrix, and the plurality of submatrices.

3. The processing apparatus according to claim 1 or 2, characterized in that, The central processing unit is also configured to generate pixel coordinates corresponding to each pixel point in the depth image based on the depth image; The memory is also used to transfer the pixel coordinates between the central processing unit and the artificial intelligence processor.

4. The processing apparatus according to any one of claims 1-3, characterized in that, The artificial intelligence processor is used for: Generate an abscissa matrix and a ordinate matrix based on the pixel coordinates; A first multiplication-addition operation is performed based on the horizontal coordinate matrix, the vertical coordinate matrix, and the plurality of submatrices to obtain the second operation result; The second multiplication-addition operation is performed based on the second calculation result, the second matrix, and the depth image to obtain the point cloud data.

5. The processing apparatus according to any one of claims 1-4, characterized in that, The processing device is a chip, and the artificial intelligence processor, the central processing unit, and the memory are integrated within the chip.

6. The processing apparatus according to any one of claims 1-5, characterized in that, The depth image is stored in the memory as image blocks and transferred to the artificial intelligence processor as image blocks.

7. A processing method, characterized in that, The method includes: The central processing unit performs matrix operations based on the camera's extrinsic and intrinsic parameter matrices to obtain a first operation result; wherein, the extrinsic parameter matrix is ​​the transformation matrix for transforming pixels in the depth image from the world coordinate system to the camera coordinate system; and the intrinsic parameter matrix is ​​the transformation matrix for transforming pixels in the depth image from the camera coordinate system to the pixel coordinate system. The memory transfers the first computation result between the central processing unit and the artificial intelligence processor; The artificial intelligence processor acquires a depth image and the pixel coordinates corresponding to the depth image, and generates point cloud data based on the depth image, the pixel coordinates, and the first calculation result; The memory transfers the point cloud data between the artificial intelligence processor and the central processing unit.

8. The method according to claim 7, characterized in that, The artificial intelligence processor acquires a depth image and the corresponding pixel coordinates of the depth image, and generates point cloud data based on the depth image, the pixel coordinates, and the first calculation result, including: The artificial intelligence processor splits the first matrix into multiple sub-matrices; The artificial intelligence processor performs multiplication and addition operations on the depth image, the pixel coordinates, the second matrix, and the plurality of submatrices to generate the point cloud data.

9. The method according to claim 7 or 8, characterized in that, The method further includes: The central processing unit generates the pixel coordinates corresponding to the depth image based on the depth image; The memory transfers the pixel coordinates between the central processing unit and the artificial intelligence processor.

10. The method according to any one of claims 7-9, characterized in that, The artificial intelligence processor generates point cloud data based on the depth image, the pixel coordinates, and the first calculation result, including: The artificial intelligence processor generates a horizontal coordinate matrix and a vertical coordinate matrix based on the pixel coordinates; The artificial intelligence processor performs a first multiplication-addition operation based on the horizontal coordinate matrix, the vertical coordinate matrix, and the plurality of submatrices to obtain a second operation result; The artificial intelligence processor performs a second multiply-add operation based on the second calculation result, the second matrix, and the depth image to obtain the point cloud data.

11. The method according to any one of claims 7-10, characterized in that, The method is applied to a processing device, which is a chip, and the artificial intelligence processor, the central processing unit, and the memory are integrated within the chip.

12. The method according to any one of claims 7-11, characterized in that, The depth image is stored in the memory as image blocks and transferred to the artificial intelligence processor as image blocks.

13. A mobile robot, characterized in that, Includes a camera and a processing device connected to the camera; the processing device includes a central processing unit, an artificial intelligence processor, and a memory. The central processing unit is used to perform matrix operations based on the camera's extrinsic and intrinsic parameter matrices to obtain a first operation result; the extrinsic parameter matrix is ​​the transformation matrix for transforming pixels in the depth image from the world coordinate system to the camera coordinate system; the intrinsic parameter matrix is ​​the transformation matrix for transforming pixels in the depth image from the camera coordinate system to the pixel coordinate system. An artificial intelligence processor is used to acquire a depth image and the pixel coordinates corresponding to the depth image, and generate point cloud data based on the depth image, the pixel coordinates and the first calculation result; A memory, coupled to the central processing unit and the artificial intelligence processor, is used to transfer the first calculation result and the point cloud data between the central processing unit and the artificial intelligence processor.

14. The mobile robot according to claim 13, characterized in that, The mobile robot further includes: a controller connected to the processing device, and a gyroscope and an accelerometer connected to the controller; The controller is used to acquire the angular velocity detected by the gyroscope and the acceleration detected by the accelerometer; The controller is also configured to control the movement direction and speed of the mobile robot based on the angular velocity, the acceleration, and the point cloud data.

Images