Communication method and device

Abstract

The invention provides a communication method and device, belongs to the technical field of communication, and is used for improving multi-mode sensing performance. In the method, after a first device obtains respective neighborhood matching degrees of N pieces of point cloud data, noise in the N pieces of point cloud data can be determined according to the respective neighborhood matching degrees of the N pieces of point cloud data; the neighborhood matching degree of the ith point cloud data in the N point cloud data is determined according to the image similarity degree of a first neighborhood i and a second neighborhood i, the first neighborhood i is a region after the ith point cloud data is projected to the first image, and the second neighborhood i is a region after the ith point cloud data is projected to the second image; the first image and the second image are obtained by performing two-dimensional perception on the target object, the first image is different from the second image, N is an integer greater than 1, and i is an integer traversing 1 to N. In this way, noise in the N pieces of point cloud data can be effectively removed, and then the multi-mode sensing performance can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of communication, and in particular to a communication method and device. BACKGROUND

[0002] Perception can be divided into two categories of two-dimensional (2D) or three-dimensional (3D) perception. Among them, 2D perception can capture a scene onto a two-dimensional plane to generate a plane image. Common 2D perception schemes include optical perception, radar perception, etc. 3D perception can capture the shape and structure of the scene in three-dimensional space, that is, generate a stereoscopic image. Common 3D perception schemes include radio frequency perception, computer tomography, etc.

[0003] At present, 2D and 3D perception schemes can be integrated to improve perception performance through multi-dimensional data, that is, multi-modal fusion perception. For example, radio frequency perception and optical perception multi-modal information can be integrated to improve the perception performance. On this basis, how to improve the multi-modal perception performance is a hot issue currently discussed. SUMMARY

[0004] Embodiments of the present application provide a communication method and device to improve multi-modal perception performance.

[0005] To achieve the above object, the technical scheme is as follows:

[0006] In a first aspect, a communication method is provided. The method is applied to a first device, for example, can be executed by the first device, can be executed by a component of the first device, such as a processor, a chip, a chip system, or a circuit of the first device, or can be implemented by a logic module or software that can realize all or part of the function of the first device. The following takes the method executed by the first device as an example for description. The method comprises: obtaining a neighborhood matching degree of each of N point cloud data, and determining noise in the N point cloud data according to the neighborhood matching degree of each of the N point cloud data; wherein N is an integer greater than 1, i is an integer traversing 1 to N, the N point cloud data is three-dimensional point cloud data obtained by perceiving a target object, and the neighborhood matching degree of the i-th point cloud data in the N point cloud data is determined according to the image similarity between a first neighborhood i and a second neighborhood i. The first neighborhood i is a region after the i-th point cloud data is projected to a first image, the second neighborhood i is a region where the i-th point cloud data is located after being projected to a second image, the first image and the second image are obtained by two-dimensional perception of the target object, and the first image is different from the second image.

[0007] Based on the method of the first aspect, after the real point cloud data (i.e., the point cloud data excluding the noise) in the N point cloud data is projected into the first image and the second image, the first neighborhood and the second neighborhood corresponding to the point cloud data correspond to the imaging results of the similar (or the same) area of the target object, which can be understood as including the point cloud data and the area with small differences, i.e., the first neighborhood and the second neighborhood have a high degree of similarity. After the noise point cloud data in the N point cloud data is projected into the first image and the second image, the first neighborhood and the second neighborhood corresponding to the noise point cloud data correspond to the imaging results of different areas, i.e., the first neighborhood and the second neighborhood have a low degree of similarity. In this way, whether the ith point cloud data is real point cloud data or not, or whether the ith point cloud data is noise, can be determined according to the degree of similarity of the first neighborhood i and the second neighborhood i, so that the noise in the N point cloud data can be effectively filtered or removed, and the multi-modal fusion perception using the real point cloud data can be realized, thereby improving the multi-modal perception performance.

[0008] It can be understood that the above-mentioned two dimensions can also be referred to as 2 dimensions, and the above-mentioned three dimensions can also be referred to as 3 dimensions.

[0009] In a possible design, the obtaining of the neighborhood matching degree of each of the N point cloud data includes: obtaining first information, the first information being used to indicate the related information of the pixel points in the first neighborhood i and the second neighborhood i; and determining the neighborhood matching degree of the ith point cloud data according to the first information. The related information of the pixel points in the first neighborhood i and the second neighborhood i can represent the image features of the first neighborhood i and the second neighborhood i, and therefore, the first device can accurately determine the neighborhood matching degree of the ith point cloud data by using the related information of the pixel points in the first neighborhood i and the second neighborhood i.

[0010] Optionally, the obtaining of the first information includes: sending a first message to the second device, the first message being used to request the related information of the pixel points in the area where the N point cloud data is located after being projected into the two-dimensional image, the two-dimensional image being obtained by sensing the target object; and receiving the first information from the second device. That is, the first device can request the related information of the pixel points in the area where the N point cloud data is located after being projected into the two-dimensional image from the second device, i.e., the first device can obtain the related information from the second device. In this way, the first device can avoid calculating the first information, thereby reducing the calculation overhead of the first device.

[0011] Further, the first message comprises neighborhood description information, the neighborhood description information being used to indicate a size and / or shape of a region where the N pieces of point cloud data are located after being projected into the two-dimensional images. That is, the second device can determine the first neighborhood i and the second neighborhood i according to the neighborhood description information after projecting the ith piece of point cloud data into the first image and the second image, and then obtain the related information of the pixel points in the first neighborhood i and the second neighborhood i. It can be understood that the first device and the second device can also pre-agree or pre-define the size and / or shape of the region where the point cloud data is located after being projected into the two-dimensional image, without limitation.

[0012] Further, the method of the first aspect further comprises: receiving a fifth capability parameter from the second device, the fifth capability parameter being used to indicate at least one of the following information of the second device: information protection degree information, perception capability information, communication capability information, computing capability information, storage capability information, internal and external parameter transmission information, neighborhood extraction and acquisition information, neighborhood feature acquisition information, or neighborhood matching degree acquisition information.

[0013] The information protection degree information is used to indicate an information exposure degree supported by the second device; the perception capability information is used to indicate a perception-related parameter of the second device; the communication capability information is used to describe whether the second device has a data sharing capability and a strength of the capability; the computing capability information is used to indicate whether the second device has a data analysis and computing capability and a strength of the capability; the internal and external parameter transmission information is used to indicate whether the second device supports transmission of internal and external parameters (i.e., internal parameters and external parameters) and two-dimensional images; the neighborhood extraction and acquisition information is used to indicate whether the second device supports neighborhood extraction and transmission of information of the extracted neighborhood; the neighborhood feature acquisition information is used to indicate whether the second device supports neighborhood feature extraction and transmission of the extracted neighborhood features; and the neighborhood matching degree acquisition information is used to indicate whether the second device supports providing a neighborhood matching degree to other devices (such as the first device). The second device indicates, through the fifth capability parameter, a capability possessed by the second device (or a device where the second device is located), so that the first device can determine an operation of the second device in the multi-modal perception based on the fifth capability parameter.

[0014] Optionally, the first information comprises first neighborhood information i and second neighborhood information i, the first neighborhood information i comprising pixel values corresponding to pixel points in the first neighborhood i, and the second neighborhood information i comprising pixel values corresponding to pixel points in the second neighborhood i; or the first information comprises first neighborhood features i and second neighborhood features i, the first neighborhood features i being used to indicate image features of the first neighborhood i, and the second neighborhood features i being used to indicate image features of the second neighborhood i, the image features being related to the pixel points. The first information can also be other information, which can be flexibly set according to actual conditions, without limitation.

[0015] Further, the method of the first aspect further includes: receiving a fifth capability parameter from the second device, the fifth capability parameter being used to indicate at least one of the following information of the second device: information protection degree information, perception capability information, communication capability information, computing capability information, storage capability information, internal-external parameter transmission information, neighborhood extraction acquisition information, neighborhood feature acquisition information, or neighborhood matching degree acquisition information.

[0016] Further, the method of the first aspect further includes: receiving a first capability parameter from the second device, the first capability parameter being used to indicate that the second device supports providing pixel values corresponding to pixel points in a first region to the first device, the first region being a region where the point cloud data is projected to a two-dimensional image; and sending a first message to the second device, including: sending the first message to the second device according to the first capability parameter. That is, the first device can determine, according to the first capability parameter, that the second device (or the device where the second device is located) has the capability to provide the first device with pixel values corresponding to pixel points in the first region. In this way, the first device can avoid failure to request the second device to send related information in the region where N point cloud data is projected to a two-dimensional image, resulting in additional communication overhead.

[0017] Further, the first capability parameter is further used to indicate at least one of the following information of the second device: information protection degree information, perception capability information, communication capability information, computing capability information, storage capability information, internal-external parameter transmission information, neighborhood feature acquisition information, or neighborhood matching degree acquisition information. In this way, the first device can determine the capabilities possessed by the second device (or the device where the second device is located) based on the first capability parameter, and thus determine the operation of the second device in multi-modal perception based on the capabilities.

[0018] Further, in the case where the first information includes the first neighborhood information i and the second neighborhood information i, determining the neighborhood matching degree of the i-th point cloud data according to the first information includes: determining the neighborhood matching degree of the i-th point cloud data according to the first neighborhood information i, the second neighborhood information i, and a matching degree calculation method, the matching degree calculation method being any one of: normalized square difference, cumulative density function, or neighborhood feature calculation. In this way, the first device can accurately determine the neighborhood matching degree of the i-th point cloud data. It can be understood that the matching degree calculation method can also be other ways, which are not limited.

[0019] Further, in a case that the first information comprises the first neighborhood feature i and the second neighborhood feature i, the method of the first aspect further comprises: receiving a second capability parameter from the second device, the second capability parameter being used to indicate that the second device supports providing the neighborhood feature to the first device, the neighborhood feature being used to indicate the image feature of the region where the point cloud data is located after being projected to the two-dimensional image; and sending the first message to the second device, comprising: sending the first message to the second device according to the second capability parameter. That is, the first device can determine that the second device (or the device where the second device is located) has the capability of providing the neighborhood feature to the first device according to the second capability parameter. In this way, the failure of the first device to request the second device to send the image feature corresponding to the region where the N point cloud data is located after being projected to the two-dimensional image can be avoided, thereby causing additional communication overhead.

[0020] Further, the second capability parameter is further used to indicate at least one of the following information of the second device: the information protection degree information, the perception capability information, the communication capability information, the calculation capability information, the storage capability information, the internal and external parameter transmission information, the neighborhood extraction acquisition information, or the neighborhood matching degree acquisition information. In this way, the first device can determine the capability possessed by the second device (or the device where the second device is located) based on the second capability parameter, and determine the operation of the second device in the multi-modal perception based on the capability.

[0021] In a possible design, the obtaining of the neighborhood matching degrees of the N point cloud data comprises: sending a second message to the second device, the second message being used to request the neighborhood matching degrees corresponding to the N point cloud data, the neighborhood matching degrees corresponding to the N point cloud data being used to indicate the image similarity degree of the region where the i-th point cloud data is located after being projected to the plurality of two-dimensional images, the plurality of two-dimensional images being obtained by perceiving the target object; and receiving the neighborhood matching degrees of the N point cloud data from the second device. That is, the first device can request the second device for the neighborhood matching degrees corresponding to the N point cloud data. In this way, the first device can avoid calculating the neighborhood matching degrees corresponding to the N point cloud data by itself, thereby reducing the calculation overhead of the first device.

[0022] Optionally, the second message comprises neighborhood description information, the neighborhood description information being used to indicate the size and / or shape of the region where the N point cloud data is located after being projected to the two-dimensional image.

[0023] Optionally, before sending the second message to the second device, the method in the first aspect further includes: receiving a third capability parameter from the second device, the third capability parameter indicating that the second device supports providing the first device with neighborhood matching degrees corresponding to the 3D point cloud data, the neighborhood matching degree corresponding to the 3D point cloud data indicating the image similarity of the regions where the 3D point cloud data is projected onto multiple 2D images; sending the second message to the second device, including: sending the second message to the second device according to the third capability parameter. That is, the first device can determine, based on the third capability parameter, that the second device (or the device to which the second device is located) has the capability to provide the first device with neighborhood matching degrees corresponding to the 3D point cloud data. This avoids the first device failing to request the second device to send neighborhood matching degrees corresponding to N point cloud data, thus preventing additional communication overhead.

[0024] Furthermore, the third capability parameter is also used to indicate at least one of the following information of the second device: information protection level information, sensing capability information, communication capability information, computing capability information, storage capability information, internal and external parameter transmission information, neighborhood extraction and acquisition information, or neighborhood feature acquisition information. In this way, the first device can determine the capabilities of the second device (or the equipment in which the second device is located) based on the third capability parameter, and thereby determine the operation of the second device in multimodal sensing based on these capabilities.

[0025] Optionally, the method in the first aspect further includes: receiving a fifth capability parameter from the second device, the fifth capability parameter being used to indicate at least one of the following information of the second device: information protection level information, sensing capability information, communication capability information, computing capability information, storage capability information, internal and external parameter transmission information, neighborhood extraction acquisition information, neighborhood feature acquisition information, or neighborhood matching degree acquisition information.

[0026] In one possible design, obtaining the neighborhood matching degree of each of the N point cloud data sets includes: acquiring parameter information, a first image, and a second image, wherein the parameter information is used to project the N point cloud data sets onto the first and second images; and determining the neighborhood matching degree of each of the N point cloud data sets based on the parameter information, the first image, and the second image. This enables the first device to accurately obtain the neighborhood matching degree of each of the N point cloud data sets.

[0027] Optionally, acquiring parameter information, the first image, and the second image includes: receiving parameter information, the first image, and the second image from the second device. The parameter information is used to instruct the second device on relevant parameters for two-dimensional perception of the target object. It is understood that the second device can periodically send relevant parameters for two-dimensional perception of the target object, as well as a two-dimensional image obtained from perceiving the target object, to the first device; alternatively, it can send relevant parameters for two-dimensional perception of the target object, as well as a two-dimensional image obtained from perceiving the target object, to the first device upon receiving a request from the first device. The specific configuration can be flexibly set according to actual circumstances and is not limited.

[0028] Furthermore, acquiring parameter information, the first image, and the second image includes: receiving first parameter information and the first image from the second device, wherein the first parameter information is used to instruct the second device to perform two-dimensional perception of the target object, and the first parameter information belongs to parameter information; and receiving second parameter information and the second image from the third device, wherein the second parameter information is used to instruct the third device to receive relevant parameters for two-dimensional perception of the target object, and the second parameter information belongs to parameter information. That is, the first device can acquire corresponding information from the second and third devices respectively. It can be understood that the second and third devices can periodically send relevant parameters for two-dimensional perception of the target object, as well as two-dimensional images obtained from perceiving the target object, to the first device; or they can send relevant parameters for two-dimensional perception of the target object, as well as two-dimensional images obtained from perceiving the target object, to the first device after receiving a request from the first device. The specific settings can be flexibly configured according to actual circumstances and are not limited.

[0029] Furthermore, the method described in the first aspect further includes: sending a third message to the second device, the third message being used to request relevant parameters and at least one image of the target object for two-dimensional perception by the second device. This enables the first device to acquire parameter information, the first image, and the second image in a timely manner.

[0030] Furthermore, before receiving the parameter information, the first image, and the second image from the second device, the method in the first aspect further includes: receiving a fourth capability parameter from the second device, the fourth capability parameter indicating that the second device supports providing the first device with relevant parameters and images for two-dimensional object perception; and sending a third message to the second device, including: sending a third message to the second device based on the fourth capability parameter. That is, the first device can determine, based on the fourth capability parameter, that the second device (or the device in which the second device is located) has the capability to provide the first device with relevant parameters and images for two-dimensional object perception, and thus request the first device to provide relevant parameters and images for two-dimensional object perception based on this capability. This avoids the first device failing to request relevant parameters and images for two-dimensional object perception from the second device, thus preventing additional communication overhead.

[0031] Furthermore, the fourth capability parameter is also used to indicate at least one of the following information of the second device: information protection level information, sensing capability information, communication capability information, computing capability information, storage capability information, neighborhood extraction acquisition information, neighborhood feature acquisition information, or neighborhood matching degree acquisition information. In this way, the first device can determine the capabilities possessed by the second device (or the device in which the second device is located) based on the third capability parameter, and thereby determine the operation of the second device in multimodal sensing based on these capabilities.

[0032] Furthermore, the relevant parameters include at least one of the following information of the second device: focal length, resolution, field of view, position, or pointing. The specific settings can be made according to the actual situation and are not limited.

[0033] Optionally, based on parameter information, the first image, and the second image, the neighborhood matching degree of each of the N point cloud data is obtained, including: projecting the i-th point cloud data onto the first image and the second image according to the parameter information; determining the first neighborhood i and the second neighborhood i based on the first image and the second image mapped with the i-th point cloud data; determining the first neighborhood information i and the second neighborhood information i based on the first neighborhood i and the second neighborhood i, where the first neighborhood information i includes the pixel value corresponding to the pixel in the first neighborhood i and the second neighborhood information i includes the pixel value corresponding to the pixel in the second neighborhood i; and determining the neighborhood matching degree of the i-th point cloud data based on the first neighborhood information i and the second neighborhood information i. In this way, the first device can accurately obtain the neighborhood matching degree of the i-th point cloud data.

[0034] Optionally, the method in the first aspect further includes: receiving a fifth capability parameter from the second device, the fifth capability parameter being used to indicate at least one of the following information of the second device: information protection level information, sensing capability information, communication capability information, computing capability information, storage capability information, internal and external parameter transmission information, neighborhood extraction acquisition information, neighborhood feature acquisition information, or neighborhood matching degree acquisition information.

[0035] In one possible design, the expression for the first neighborhood i is: Nr(u,v)={(u+x,v+y)|-r≤x≤r,-k≤y≤k}, where (u,v) are the coordinates of the point i mapped from the point cloud data to the first image, (2r+1) is greater than or equal to 1 and less than the horizontal resolution of the first image, and (2k+1) is greater than or equal to 1 and less than the vertical resolution of the first image; or, the expression for the first neighborhood i is: Nr(u,v)={(u+x,v+y)|x 2 +y 2 ≤r 2Let (u, v) be the coordinates of the i-th point cloud data mapped to the first image, where r is less than a first value, which is the minimum of the horizontal and vertical resolutions of the first image. It can be understood that | is a separator; (u+x, v+y) before | indicates the pixel coordinates of the first neighboring i, and the values ​​after | represent the range of x and y.

[0036] Secondly, a communication method is provided, which is applied to a second device. This method can be executed by the second device, such as its processor, chip, chip system, or circuit, or by a logic module or software capable of implementing all or part of the functions of the second device. The following description uses the execution of this method by a second device as an example. The method includes: receiving a first message from a first device, the first message requesting information about pixels in the region where N point cloud data are projected onto a two-dimensional image; the N point cloud data are three-dimensional point cloud data obtained by perceiving a target object, where N is an integer greater than 1; and the two-dimensional image is obtained by perceiving the target object. In response to the first message, sending first information to the first device, the first information indicating information about pixels in a first neighborhood i and a second neighborhood i, where the first neighborhood i is the region where the i-th point cloud data is projected onto the first image, and the second neighborhood i is the region where the i-th point cloud data is projected onto the second image; the first image and the second image are obtained by two-dimensional perception of the target object, and the first image and the second image are different; and i is an integer ranging from 1 to N.

[0037] In one possible design, the first message includes neighborhood description information, which indicates the size and / or shape of the region where the N point cloud data are projected onto the two-dimensional image.

[0038] In one possible design, the first information includes first neighborhood information i and second neighborhood information i, where the first neighborhood information i includes the pixel value corresponding to the pixel in the first neighborhood i, and the second neighborhood information i includes the pixel value corresponding to the pixel in the second neighborhood i; or, the first information includes first neighborhood feature i and second neighborhood feature i, where the first neighborhood feature i is used to indicate the image features of the first neighborhood i, and the second neighborhood feature i is used to indicate the image features of the second neighborhood i, and the image features are related to the pixel.

[0039] Optionally, if the first information includes first neighborhood information i and second neighborhood information i, the method in the second aspect further includes: sending a first capability parameter to the first device, the first capability parameter being used to indicate that the second device supports providing the pixel values ​​corresponding to the pixels in the first region to the first device, the first region being the region where the point cloud data is projected onto the two-dimensional image.

[0040] Furthermore, the first capability parameter is also used to indicate at least one of the following information of the second device: information protection level information, sensing capability information, communication capability information, computing capability information, storage capability information, internal and external parameter transmission information, neighborhood feature acquisition information, or neighborhood matching degree acquisition information.

[0041] Optionally, if the first information includes a first neighborhood feature i and a second neighborhood feature i, the method in the second aspect further includes: sending a second capability parameter to the first device, the second capability parameter being used to indicate that the second device supports providing neighborhood features to the first device, the neighborhood features being used to indicate the image features of the region where the point cloud data is projected onto the two-dimensional image.

[0042] Furthermore, the second capability parameter is also used to indicate at least one of the following information of the second device: information protection level information, sensing capability information, communication capability information, computing capability information, storage capability information, internal and external parameter transmission information, neighborhood extraction acquisition information, or neighborhood matching degree acquisition information.

[0043] In one possible design, the method described in the second aspect further includes: sending a fifth capability parameter to the first device, the fifth capability parameter being used to indicate at least one of the following information of the second device: information protection level information, sensing capability information, communication capability information, computing capability information, storage capability information, internal and external parameter transmission information, neighborhood extraction acquisition information, neighborhood feature acquisition information, or neighborhood matching degree acquisition information.

[0044] Furthermore, the technical effects of the method described in the second aspect can also refer to the technical effects of the method described in the first aspect, and will not be repeated here.

[0045] Thirdly, a communication method is provided, which is applied to a second device. This method can be executed by the second device itself, such as its processor, chip, chip system, or circuit, or it can be implemented by a logic module or software capable of performing all or part of the functions of the second device. The following explanation uses the execution of this method by a second device as an example. The method includes: receiving a second message from a first device, the second message being used to request neighborhood matching degrees corresponding to N point cloud data, the N point cloud data being three-dimensional point cloud data obtained by perceiving a target object, N being an integer greater than 1, the neighborhood matching degrees corresponding to the N point cloud data being used to indicate the image similarity of the region where the i-th point cloud data is projected onto multiple two-dimensional images, the multiple two-dimensional images being obtained by perceiving the target object, i being an integer from 1 to N; in response to the second message, sending the neighborhood matching degrees of each of the N point cloud data to the first device, the neighborhood matching degree of the i-th point cloud data being determined based on the image similarity between a first neighborhood i and a second neighborhood i, the first neighborhood i being the region where the i-th point cloud data is projected onto a first image, the second neighborhood i being the region where the i-th point cloud data is projected onto a second image, the first image and the second image being obtained by two-dimensional perception of the target object, the first image and the second image being different.

[0046] In one possible design, before receiving the second message from the first device, the method of the third aspect further includes: sending a third capability parameter to the first device, the third capability parameter being used to indicate that the second device supports providing the neighborhood matching degree corresponding to the point cloud data to the first device, the neighborhood matching degree corresponding to the point cloud data being used to indicate the image similarity of the region where the point cloud data is projected onto multiple two-dimensional images.

[0047] Optionally, the third capability parameter is also used to indicate at least one of the following information of the second device: information protection level information, sensing capability information, communication capability information, computing capability information, storage capability information, internal and external parameter transmission information, neighborhood extraction and acquisition information, or neighborhood feature acquisition information.

[0048] In one possible design, before receiving the second message from the first device, the method of the third aspect further includes: sending a fifth capability parameter to the first device, the fifth capability parameter being used to indicate at least one of the following information of the second device: information protection level information, sensing capability information, communication capability information, computing capability information, storage capability information, internal and external parameter transmission information, neighborhood extraction acquisition information, neighborhood feature acquisition information, or neighborhood matching degree acquisition information.

[0049] In one possible design, the second message includes neighborhood description information, which indicates the size and / or shape of the region where the N point cloud data are projected onto the two-dimensional image.

[0050] Optionally, the method in the third aspect further includes: projecting the i-th point cloud data onto a first image and a second image; determining a first neighborhood i and a second neighborhood i based on the first image and the second image mapped with the i-th point cloud data; determining first neighborhood information i and second neighborhood information i based on the first neighborhood i and the second neighborhood i, wherein the first neighborhood information i includes the pixel value corresponding to the pixel in the first neighborhood i, and the second neighborhood information i includes the pixel value corresponding to the pixel in the second neighborhood i; and determining the neighborhood matching degree of the i-th point cloud data based on the first neighborhood information i and the second neighborhood information i.

[0051] Furthermore, the technical effects of the method described in the third aspect can also refer to the technical effects of the method described in the first aspect, and will not be elaborated here.

[0052] Fourthly, a communication method is provided. This method is applied to a second device, for example, it can be executed by the second device, such as its processor, chip, chip system, or circuit, or it can be implemented by a logic module or software capable of implementing all or part of the functions of the second device. The following description uses the execution of this method by a second device as an example. The method includes: acquiring parameter information, a first image, and a second image; the parameter information is used to indicate relevant parameters for the second device to perform two-dimensional perception of a target object; the first image and the second image are obtained by the second device through two-dimensional perception of the target object, and the first image and the second image are different; and sending the parameter information, the first image, and the second image to a first device.

[0053] In one possible design, the method in the fourth aspect includes: receiving a third message from a first device, the third message being used to request relevant parameters and at least one image of a second device for two-dimensional perception of a target object; sending parameter information, a first image, and a second image to the first device, including: in response to the third message, sending parameter information, a first image, and a second image to the first device.

[0054] In one possible design, before sending parameter information, the first image, and the second image to the first device, the method in the fourth aspect further includes: sending a fourth capability parameter to the first device, the fourth capability parameter being used to instruct the second device to support providing the first device with relevant parameters and images for two-dimensional perception of objects.

[0055] Optionally, the fourth capability parameter is also used to indicate at least one of the following information of the second device: information protection level information, sensing capability information, communication capability information, computing capability information, storage capability information, neighborhood extraction acquisition information, neighborhood feature acquisition information, or neighborhood matching degree acquisition information.

[0056] In one possible design, the relevant parameters include at least one of the following information about the second device: focal length, resolution, field of view, position, or pointing.

[0057] In one possible design, before receiving the third message from the first device, the method in the fourth aspect further includes: sending a fifth capability parameter to the first device, the fifth capability parameter being used to indicate at least one of the following information of the second device: information protection level information, sensing capability information, communication capability information, computing capability information, storage capability information, internal and external parameter transmission information, neighborhood extraction acquisition information, neighborhood feature acquisition information, or neighborhood matching degree acquisition information.

[0058] Furthermore, the technical effects of the method described in the fourth aspect can also refer to the technical effects of the method described in the first aspect, and will not be repeated here.

[0059] Fifthly, a communication method is provided. This method is applied to a first device, and can be executed by the first device itself, or by a component of the first device, such as a processor, chip, chip system, or circuit. It can also be implemented by a logic module or software capable of implementing all or part of the functions of the first device. The following description uses the execution of this method by the first device as an example. The method includes: receiving capability parameters from a second device, and sending a first message to the second device according to the capability parameters; wherein the capability parameters indicate first information that the second device can report to the first device, the first information indicates relevant information for the second device to perceive an object, or information processed based on the relevant information obtained by the second device to perceive an object, and the first message requests relevant information for the second device to perceive a target object.

[0060] Based on the method in the fifth aspect, after receiving capability parameters from the second device, the first device can request relevant information from the second device regarding the second device's perception of the target object. In other words, the first device can determine the information that the second device (or the device containing the second device) can provide based on its capabilities, and then request that information. This avoids the first device failing to request information from the second device, thus preventing additional communication overhead.

[0061] In one possible design, the capability parameters include at least one of the following information of the second device: information protection level information, sensing capability information, communication capability information, computing capability information, storage capability information, internal and external parameter transmission information, neighborhood acquisition information, or neighborhood matching degree acquisition information.

[0062] Among them, information protection level information is used to indicate the information exposure level supported by the second device; perception capability information is used to indicate the perception-related parameters of the second device; communication capability information is used to describe whether the second device has data sharing capabilities and the strength of such capabilities; computing capability information is used to indicate whether the second device has data analysis and computing capabilities and the strength of such capabilities; internal and external parameter transmission information is used to indicate whether the second device supports the transmission of internal and external parameters (i.e., internal parameters and external parameters) and two-dimensional images; neighborhood acquisition information is used to indicate whether the second device supports providing the first device with relevant information about pixels in a first region, which is the region where the point cloud data is projected onto the two-dimensional image; and neighborhood matching degree acquisition information is used to indicate whether the second device supports providing neighborhood matching degrees to other devices (such as the first device). The second device indicates the capabilities possessed by the second device (or the device on which the second device is located) to the first device through capability parameters, enabling the first device to determine the operation of the second device in multimodal perception based on these capability parameters.

[0063] Optionally, the capability parameters include neighborhood acquisition information, which indicates that the second device supports providing the first device with relevant information about pixels in a first region, where the first region is the area where the point cloud data is projected onto a 2D image; and a first message, which requests relevant information for the second device to perceive a target object, including: the first message requests relevant information about pixels in the area where N point cloud data is projected onto a 2D image, where N point cloud data are 3D point cloud data obtained from perceiving the target object, N is an integer greater than 1, and the 2D image is obtained from perceiving the target object. That is, when the second device (or the device containing the second device) has the capability to provide the first device with relevant information about pixels in the first region, the first device can request relevant information about pixels in the area where the N point cloud data is projected onto a 2D image from the second device. This ensures that the first device obtains relevant information about pixels in the area where the N point cloud data is projected onto a 2D image from the second device, avoiding request failure.

[0064] Furthermore, the first message includes neighborhood description information, which indicates the size and / or shape of the region where the N point cloud data are projected onto the two-dimensional image. Thus, after projecting the i-th point cloud data from the N point cloud data onto the first and second images, the second device can determine the first neighborhood i and the second neighborhood i based on the neighborhood description information, and then obtain relevant information about the pixels in the first and second neighborhoods i, where i is an integer from 1 to N. It is understood that the first and second devices can also pre-agree or pre-define the size and / or shape of the region where the point cloud data is projected onto the two-dimensional image, without restriction.

[0065] Furthermore, the method in the fifth aspect further includes: receiving second information from the second device, the second information being used to indicate relevant information about pixels in the first neighborhood i and the second neighborhood i, the first neighborhood i being the region where the i-th point cloud data in N point cloud data is projected onto the first image, the second neighborhood i being the region where the i-th point cloud data is projected onto the second image, the first image and the second image being images obtained by two-dimensional perception of the target object, the first image being different from the second image, and i being an integer from 1 to N; obtaining the neighborhood matching degree of the i-th point cloud data according to the second information; and determining the noise in the N point cloud data according to the neighborhood matching degree of the i-th point cloud data.

[0066] Furthermore, the second information includes first neighborhood information i and second neighborhood information i. First neighborhood information i includes the pixel value corresponding to the pixel in first neighborhood i, and second neighborhood information i includes the pixel value corresponding to the pixel in second neighborhood i. Alternatively, the second information includes first neighborhood feature i and second neighborhood feature i. First neighborhood feature i indicates the image features of first neighborhood i, and second neighborhood feature i indicates the image features of second neighborhood i. The image features are related to the pixel. The second information can also be other information, and can be flexibly set according to the actual situation without limitation.

[0067] Furthermore, when the second information includes the first neighborhood information i and the second neighborhood information i, the method in the fifth aspect further includes: obtaining the neighborhood matching degree of the i-th point cloud data based on the first neighborhood information i, the second neighborhood information i, and the matching degree calculation method, wherein the matching degree calculation method is any one of the following: normalized squared difference, cumulative density function, or neighborhood feature calculation. In this way, the first device can accurately determine the neighborhood matching degree of the i-th point cloud data. It is understood that the matching degree calculation method can also be other methods, without limitation.

[0068] Furthermore, when the second information includes first neighborhood information i and second neighborhood information i, the neighborhood acquisition information is used to indicate that the second device supports providing the first device with relevant information about pixels in the first region, including: neighborhood extraction acquisition information is used to indicate that the second device supports providing the first device with pixel values ​​corresponding to pixels in the first region. In this case, the capability parameter may also include neighborhood feature acquisition information, which is used to indicate whether the second device supports neighborhood feature extraction and the transmission of extracted neighborhood features. The specific settings can be configured according to actual circumstances and are not limited.

[0069] Furthermore, when the second information includes the first neighborhood feature i and the second neighborhood feature i, the neighborhood acquisition information is used to indicate that the second device supports providing the first device with relevant information about pixels in the first region, including: neighborhood extraction acquisition information is used to indicate that the second device supports providing the first device with image features of the first region. In this case, the capability parameter may also include neighborhood extraction acquisition information, which is used to indicate whether the second device supports neighborhood extraction and the transmission of extracted neighborhood information. The specific settings can be configured according to actual circumstances and are not limited.

[0070] Optionally, the capability parameters include neighborhood matching degree acquisition information, which is used to indicate that the second device supports providing the neighborhood matching degree corresponding to the point cloud data to the first device. The neighborhood matching degree corresponding to the point cloud data is used to indicate the image similarity of the region where the point cloud data is projected onto multiple two-dimensional images. The first message is used to request relevant information for the second device to perceive the target object, including: the first message is used to request the neighborhood matching degree corresponding to N point cloud data, where N point cloud data are three-dimensional point cloud data obtained by perceiving the target object, and N is an integer greater than 1. The neighborhood matching degree corresponding to the N point cloud data is used to indicate the image similarity of the region where the i-th point cloud data in the N point cloud data is projected onto multiple two-dimensional images, where i is an integer from 1 to N, and the multiple two-dimensional images are obtained by perceiving the target object. That is, when the second device (or the device in which the second device is located) has the capability to provide the neighborhood matching degree corresponding to the point cloud data to the first device, the first device can request the neighborhood matching degree corresponding to the N point cloud data from the second device. In this way, it can be ensured that the first device obtains the neighborhood matching degree corresponding to N point cloud data from the second device, thus avoiding the failure of the first device's request.

[0071] Furthermore, the first message includes neighborhood description information, which indicates the size or shape of the region where the N point cloud data are projected onto the two-dimensional image.

[0072] Furthermore, the method in the fifth aspect also includes: receiving the neighborhood matching degree of each of the N point cloud data from the second device, wherein the neighborhood matching degree of the i-th point cloud data is determined based on the image similarity between the first neighborhood i and the second neighborhood i, where the first neighborhood i is the region where the i-th point cloud data is projected onto the first image, and the second neighborhood i is the region where the i-th point cloud data is projected onto the second image, and the first image and the second image are images obtained by two-dimensional perception of the target object, and the first image and the second image are different; and determining the noise in the N point cloud data based on the neighborhood matching degree of each of the N point cloud data. It can be understood that after determining the noise in the N point cloud data, the first device can filter or remove the noise, thereby using the real point cloud data (the point cloud data in the N point cloud data excluding the noise) for multimodal perception, thereby improving the multimodal perception performance.

[0073] Optionally, the capability parameters include intrinsic and extrinsic parameter transmission information, which instructs the second device to support providing the first device with relevant parameters and images for object perception. The first message requests relevant information from the second device for object perception, including: the first message requests relevant parameters and at least one image from the second device for object perception. That is, when the second device (or the device containing the second device) has the relevant parameters and images for (two-dimensional) object perception to provide to the first device, the first device can request relevant information from the second device for object perception. This ensures that the first device obtains the relevant information from the second device for object perception, preventing request failures.

[0074] Furthermore, the relevant parameters include at least one of the following information of the second device: focal length, resolution, field of view, position, or pointing. The specific settings can be made according to the actual situation and are not limited.

[0075] Furthermore, the method in the fifth aspect further includes: receiving parameter information, a first image, and a second image from the second device, wherein the parameter information is used to project N point cloud data onto the first image and the second image, the N point cloud data being three-dimensional point cloud data obtained by perceiving the target object, N being an integer greater than 1, the first image and the second image being obtained by two-dimensional perception of the target object, and the first image being different from the second image; obtaining the neighborhood matching degree of each of the N point cloud data according to the parameter information, the first image, and the second image, wherein the neighborhood matching degree of the i-th point cloud data is determined based on the image similarity between the first neighborhood i and the second neighborhood i, the first neighborhood i being the region where the i-th point cloud data is projected onto the first image, the second neighborhood i being the region where the i-th point cloud data is projected onto the second image, i being an integer from 1 to N; and determining the noise in the N point cloud data according to the neighborhood matching degree of each of the N point cloud data.

[0076] Furthermore, based on the parameter information, the first image, and the second image, the neighborhood matching degree of each of the N point cloud data is obtained, including: projecting the i-th point cloud data onto the first image and the second image according to the parameter information; determining the first neighborhood i and the second neighborhood i based on the first image and the second image mapped with the i-th point cloud data; determining the first neighborhood information i and the second neighborhood information i based on the first neighborhood i and the second neighborhood i, where the first neighborhood information i includes the pixel value corresponding to the pixel in the first neighborhood i and the second neighborhood information i includes the pixel value corresponding to the pixel in the second neighborhood i; and determining the neighborhood matching degree of the i-th point cloud data based on the first neighborhood information i and the second neighborhood information i. In this way, the first device can accurately obtain the neighborhood matching degree of the i-th point cloud data.

[0077] Sixthly, a communication method is provided, which is applied to a second device. This method can be executed by the second device itself, such as its processor, chip, chip system, or circuit, or by a logic module or software capable of implementing all or part of the functions of the second device. The following description uses the execution of this method by a second device as an example. The method includes: acquiring capability parameters and sending the capability parameters to a first device, wherein the capability parameters indicate first information that the second device can report to the first device, and the first information indicates relevant information for the second device to perceive an object, or information processed based on the relevant information obtained by the second device to perceive the object.

[0078] In one possible design, the capability parameters include at least one of the following information of the second device: information protection level information, sensing capability information, communication capability information, computing capability information, storage capability information, internal and external parameter transmission information, neighborhood acquisition information, or neighborhood matching degree acquisition information.

[0079] Optionally, the capability parameters include neighborhood acquisition information, which is used to indicate that the second device supports providing the first device with relevant information about pixels in the first region, where the first region is the region where the point cloud data is projected onto the two-dimensional image.

[0080] Furthermore, the method in the sixth aspect further includes: receiving a first message from the first device, the first message being used to request information about pixels in the region where N point cloud data are projected onto a two-dimensional image, the N point cloud data being three-dimensional point cloud data obtained by perceiving the target object, N being an integer greater than 1, and the two-dimensional image being obtained by perceiving the target object; responding to the first message, sending second information to the first device, the second information being used to indicate information about pixels in a first neighborhood i and a second neighborhood i, the first neighborhood i being the region where the i-th point cloud data in the N point cloud data is projected onto the first image, the second neighborhood i being the region where the i-th point cloud data is projected onto the second image, the first image and the second image being images obtained by two-dimensional perception of the target object, the first image being different from the second image, and i being an integer ranging from 1 to N.

[0081] Furthermore, the first message includes neighborhood description information, which indicates the size and / or shape of the region where the N point cloud data are projected onto the two-dimensional image.

[0082] Furthermore, the second information includes first neighborhood information i and second neighborhood information i, where first neighborhood information i includes the pixel value corresponding to the pixel in first neighborhood i, and second neighborhood information i includes the pixel value corresponding to the pixel in second neighborhood i; or, the second information includes first neighborhood feature i and second neighborhood feature i, where first neighborhood feature i is used to indicate the image features of first neighborhood i, and second neighborhood feature i is used to indicate the image features of second neighborhood i, and the image features are related to pixel values.

[0083] Furthermore, when the second information includes the first neighborhood information i and the second neighborhood information i, the neighborhood acquisition information is used to instruct the second device to support providing the first device with relevant information of the pixels in the first region, including: the neighborhood extraction acquisition information is used to instruct the second device to support providing the first device with the pixel values ​​corresponding to the pixels in the first region.

[0084] Furthermore, when the second information includes the first neighborhood feature i and the second neighborhood feature i, the neighborhood acquisition information is used to instruct the second device to support providing the first device with relevant information about the pixels in the first region, including: the neighborhood extraction acquisition information is used to instruct the second device to support providing the first device with image features of the first region.

[0085] Optionally, the capability parameters include neighborhood matching degree acquisition information, which is used to indicate that the second device supports providing the neighborhood matching degree corresponding to the point cloud data to the first device, and the neighborhood matching degree corresponding to the point cloud data is used to indicate the image similarity of the region where the point cloud data is projected onto multiple two-dimensional images.

[0086] Furthermore, the method in the sixth aspect further includes: receiving a first message from the first device, the first message being used to request the neighborhood matching degree corresponding to N point cloud data, the N point cloud data being three-dimensional point cloud data obtained by perceiving the target object, N being an integer greater than 1, the neighborhood matching degree corresponding to the N point cloud data being used to indicate the image similarity of the region where the i-th point cloud data in the N point cloud data is projected onto multiple two-dimensional images, i being an integer from 1 to N, and the multiple two-dimensional images being obtained by perceiving the target object; in response to the first message, sending the neighborhood matching degree of each of the N point cloud data to the first device, the neighborhood matching degree of the i-th point cloud data in the N point cloud data being determined based on the image similarity between the first neighborhood i and the second neighborhood i, the first neighborhood i being the region where the i-th point cloud data is projected onto the first image, the second neighborhood i being the region where the i-th point cloud data is projected onto the second image, the first image and the second image being images obtained by two-dimensional perception of the target object, and the first image and the second image being different.

[0087] Furthermore, the first message includes neighborhood description information, which indicates the size or shape of the region where the N point cloud data are projected onto the two-dimensional image.

[0088] Furthermore, the method in the sixth aspect further includes: projecting the i-th point cloud data onto a first image and a second image; determining a first neighborhood i and a second neighborhood i based on the first image and the second image mapped with the i-th point cloud data; determining first neighborhood information i and second neighborhood information i based on the first neighborhood i and the second neighborhood i, wherein the first neighborhood information i includes the pixel value corresponding to the pixel in the first neighborhood i, and the second neighborhood information i includes the pixel value corresponding to the pixel in the second neighborhood i; and determining the neighborhood matching degree of the i-th point cloud data based on the first neighborhood information i and the second neighborhood information i.

[0089] Optionally, the capability parameters include intrinsic and extrinsic parameter transmission information, which is used to indicate that the second device supports providing the first device with relevant parameters and images for object perception.

[0090] Furthermore, the relevant parameters include at least one of the following information about the second device: focal length, resolution, field of view, position, or pointing.

[0091] Furthermore, the method in the sixth aspect further includes: receiving a first message from a first device, the first message being used to request relevant parameters and at least one image from a second device to perceive a target object; in response to the first message, sending parameter information, a first image, and a second image to the first device, the parameter information being used to project N point cloud data onto the first image and the second image, the N point cloud data being three-dimensional point cloud data obtained by perceiving the target object, N being an integer greater than 1, the first image and the second image being obtained by two-dimensional perception of the target object, and the first image being different from the second image.

[0092] Furthermore, the technical effects of the method described in the sixth aspect can also refer to the technical effects of the method described in the fifth aspect, and will not be repeated here.

[0093] A seventh aspect provides a communication method, the method comprising: a first device performing the method of the first aspect, and a second device performing the method of the second aspect; or, the first device performing the method of the first aspect, and the second device performing the method of the third aspect; or, the first device performing the method of the first aspect, and the second device performing the method of the fourth aspect.

[0094] Eighthly, a communication method is provided, the method comprising: a first device performing the method described in the fifth aspect, and a second device performing the method described in the sixth aspect.

[0095] A ninth aspect provides a communication device. The communication device includes: modules or units (e.g., chips, chip systems, or circuits) for performing each of the methods / operations / steps / actions described in any one of the first to sixth aspects, such as a transceiver module and a processing module. For example, the transceiver module is used to indicate the transceiver function of the communication device, and the processing module is used to perform functions of the communication device other than the transceiver function.

[0096] Optionally, the transceiver module may include a transmitting module and a receiving module. The transmitting module implements the transmitting function of the communication device described in the ninth aspect, and the receiving module implements the receiving function of the communication device described in the ninth aspect.

[0097] Optionally, the communication device according to the ninth aspect may further include a storage module storing a program or instructions. When the processing module executes the program or instructions, the communication device can perform the method described in any one of the first to sixth aspects.

[0098] It is understood that the communication device described in the ninth aspect may be a terminal device or a network device, or a chip (system) or other component or assembly that can be disposed in the terminal device or the network device, or a device that includes the terminal device or the network device. This application does not limit it in this regard.

[0099] Furthermore, the technical effects of the communication device described in the ninth aspect can be referred to the technical effects of the method described in any of the implementations of the first to sixth aspects, and will not be repeated here.

[0100] A tenth aspect provides a communication device. The communication device includes a processor, which, when executing computer instructions, causes the communication device to perform the method described in any one of the possible implementations of the first to sixth aspects.

[0101] In one possible design, the communication device described in the tenth aspect may further include a transceiver. The transceiver may be a transceiver circuit or an interface circuit. The transceiver can be used for communication between the communication device described in the tenth aspect and other communication devices.

[0102] In one possible design, the communication device described in the tenth aspect may further include a memory. This memory may be integrated with the processor or disposed separately. The memory may be used to store computer programs and / or data relating to the methods described in any of the first to sixth aspects.

[0103] In the embodiments of this application, the communication device described in the tenth aspect may be a terminal device or network device described in any one of the first to sixth aspects, or may be a chip (system) or other component or assembly disposed in the terminal device or the network device, or may include the terminal device or the network device.

[0104] Furthermore, the technical effects of the communication device described in the tenth aspect can be referred to the technical effects of the method described in any of the implementations of the first to sixth aspects, and will not be repeated here.

[0105] Eleventhly, a communication device is provided. The communication device includes a processor coupled to a memory, the processor being configured to execute a computer program stored in the memory, such that the communication device performs the method described in any one of the possible implementations of the first to sixth aspects.

[0106] In one possible design, the communication device described in the eleventh aspect may further include a transceiver. This transceiver may be a transceiver circuit or an interface circuit. The transceiver can be used for communication between the communication device described in the eleventh aspect and other communication devices.

[0107] In the embodiments of this application, the communication device described in the eleventh aspect may be a terminal device or network device described in any one of the first to sixth aspects, or may be a chip (system) or other component or assembly disposed in the terminal device or network device, or may include the terminal device or network device.

[0108] Furthermore, the technical effects of the communication device described in the eleventh aspect can be referred to the technical effects of the method described in any of the implementations of the first to sixth aspects, and will not be repeated here.

[0109] In a twelfth aspect, a communication device is provided, comprising: a processor and a memory; the memory is configured to store a computer program, which, when executed by the processor, causes the communication device to perform the method described in any one of the first to sixth aspects.

[0110] In one possible design, the communication device described in the twelfth aspect may further include a transceiver. The transceiver may be a transceiver circuit or an interface circuit. The transceiver can be used for communication between the communication device described in the twelfth aspect and other communication devices.

[0111] In the embodiments of this application, the communication device described in the twelfth aspect may be a terminal device or network device described in any one of the first to sixth aspects, or may be a chip (system) or other component or assembly disposed in the terminal device or the network device, or may include the terminal device or the network device.

[0112] Furthermore, the technical effects of the communication device described in the twelfth aspect can be referred to the technical effects of the method described in any of the implementations of the first to sixth aspects, and will not be repeated here.

[0113] In a thirteenth aspect, a communication device is provided for implementing the method described in any of the possible implementations of the first to sixth aspects.

[0114] In a fourteenth aspect, a communication chip is provided, comprising: a logic circuit and a communication interface, the logic circuit being used to execute computer instructions, and the communication interface being used for the communication chip to communicate with other devices or chips, wherein when the logic circuit executes the computer instructions, the method described in any one of the first to sixth aspects is implemented.

[0115] In a fifteenth aspect, a communication system is provided, comprising at least one of the following: a first means for performing the method of the first aspect, or a second means for performing the method of the second aspect; or, a first means for performing the method of the first aspect, or a second means for performing the method of the third aspect; or, the communication system comprises at least one of the following: a first means for performing the method of the first aspect, or a second means for performing the method of the fourth aspect.

[0116] In a sixteenth aspect, a communication system is provided, comprising at least one of the following: a first means for performing the method of the fifth aspect, or a second means for performing the method of the sixth aspect.

[0117] In a seventeenth aspect, a computer-readable storage medium is provided, comprising: a computer program or instructions; when the computer program or instructions are executed on a computer, causing the computer to perform the method described in any one of the possible implementations of the first to sixth aspects.

[0118] Eighteenth aspect: A computer program product is provided, including a computer program or instructions that, when run on a computer, cause the computer to perform the method described in any one of the possible implementations of the first to sixth aspects. Attached Figure Description

[0119] Figure 1 A schematic diagram of the optical sensing coordinate system provided in an embodiment of this application;

[0120] Figure 2 This is a schematic diagram illustrating the camera coordinate system to image coordinate system provided in an embodiment of this application.

[0121] Figure 3 This is a schematic diagram illustrating the relationship between the image coordinate system and the pixel coordinate system provided in an embodiment of this application.

[0122] Figure 4 A schematic diagram illustrating the relationship between field of view and focal length provided in an embodiment of this application;

[0123] Figure 5 A schematic diagram illustrating the radio frequency sensing results provided in an embodiment of this application;

[0124] Figure 6 A framework diagram of the multimodal fusion sensing scheme provided in the embodiments of this application;

[0125] Figure 7 A schematic diagram of the communication method provided in the embodiments of this application. Figure 1 ;

[0126] Figure 8 A schematic diagram of neighborhood matching provided for an embodiment of this application;

[0127] Figure 9 A schematic diagram showing the projection of real point cloud data and noisy point cloud data provided in the embodiments of this application;

[0128] Figure 10 A schematic diagram illustrating the simulation configuration and results provided in the embodiments of this application;

[0129] Figure 11 A schematic diagram of the communication method provided in the embodiments of this application.Figure 2 ;

[0130] Figure 12 A schematic diagram of the communication method provided in the embodiments of this application. Figure 3 ;

[0131] Figure 13 A schematic diagram of the communication method provided in the embodiments of this application. Figure 4 ;

[0132] Figure 14 A schematic diagram of the communication method provided in the embodiments of this application. Figure 5 ;

[0133] Figure 15 A schematic diagram of the communication method provided in the embodiments of this application. Figure 6 ;

[0134] Figure 16 Schematic diagram of the communication device provided in the embodiments of this application Figure 1 ;

[0135] Figure 17 Schematic diagram of the communication device provided in the embodiments of this application Figure 2 . Detailed Implementation

[0136] For ease of understanding, the technical terms involved in the embodiments of this application will be introduced below.

[0137] 1. Perception

[0138] Perception encompasses various methods, broadly categorized into 2D and 3D perception. 2D perception captures a scene onto a two-dimensional plane, generating a planar image. This image includes width and height information, but not depth information. Common 2D perception methods include optical perception and radar perception. 3D perception captures the shape and structure of a scene in three-dimensional space, generating a stereo image. This image includes width, height, and depth information. Common 3D perception methods include radio frequency sensing and computed tomography (CT) scanning. The following sections will describe optical perception and radio frequency sensing respectively.

[0139] 1.1 Optical Sensing

[0140] Optical sensing uses image sensors to detect light waves and generate images. The parameters of an optical sensing node are divided into intrinsic parameters and extrinsic parameters. Intrinsic parameters are related to the camera's own characteristics, such as focal length, field of view, and resolution; extrinsic parameters refer to the camera's position, rotation direction, etc. The imaging process of optical sensing involves four coordinate systems (world, camera, image, and pixel coordinate systems) and the transformations between these four coordinate systems. The following section combines... Figure 1 Explain these four coordinate systems.

[0141] like Figure 2 As shown, the optical sensing coordinate system involves the world coordinate system, camera coordinate system, image coordinate system, and pixel coordinate system.

[0142] The world coordinate system is a three-dimensional coordinate system of the objective world. Placing an optical sensing node in three-dimensional space, and using the world coordinate system as a reference to describe the position of the optical sensing node, is denoted as P(X). W ,Y W Z W ).

[0143] The camera coordinate system is established with the optical sensing node (such as the optical center of the camera) as the origin and the camera's optical axis as the Z-axis. In the camera coordinate system, the position of the optical sensing node can be represented as (X... C ,Y C Z C It is understandable that the transformation from the world coordinate system to the camera coordinate system can be achieved through rotation and translation operations, and vice versa.

[0144] An image coordinate system has its origin at the center of the camera's image sensor, with the X and Y axes parallel to the two vertical sides of the image sensor, and its coordinate values ​​are represented by (x, y). Image coordinate systems typically use physical units (such as millimeters (mm)) to represent the position of pixels in the image.

[0145] The pixel coordinate system has its origin at the top-left corner of the image sensor, with the X and Y axes parallel to the X and Y axes of the image coordinate system, respectively. Its coordinate values ​​are represented by (u, v). Each image contains M rows and N columns of elements, each element being called a pixel. The pixel coordinate system uses pixels as its unit. Rows correspond to the horizontal direction of the image sensor, and columns correspond to the vertical direction.

[0146] The transformation process for the above four coordinate systems is as follows:

[0147] (1) World coordinate system → Camera coordinate system: This is a 3D to 3D projection, which transforms the object's coordinates from the world coordinate system to the camera coordinate system through rotation and translation. Camera position refers to the camera's position O in the world coordinate system.c Camera pointing refers to the direction of the three axes of the camera coordinate system (usually the x, y, and z axes) in the world coordinate system [X]. axis ,Y axis Z axis When the directions of any two coordinate axes in the camera's pointing direction are determined, the direction of the other axis can be calculated, such as Y. axis =Z axis ×X axis The rotation matrix from the world coordinate system to the camera coordinate system is a 3×3 orthogonal matrix, represented as follows: ‖·‖ is the norm of the vector. The translation vector is a 3×1 vector, represented as t=-RO. c It can be seen that the camera's rotation and translation are determined by its extrinsic parameters (such as camera position and pointing), and the extrinsic parameter matrix of the camera is defined as follows:

[0148] (2) Camera coordinate system → Image coordinate system: This is a 3D to 2D projection, which can be calculated based on the principle of similar triangles. For example... Figure 3 As shown, Where f represents the camera's focal length, expressed in matrix form as follows: During this projection process, depth information is lost. For example, as... Figure 3 As shown, points P, B, and A in the camera coordinate system will be projected onto points p, C, and o in the image. Additionally, other points on the ray Oc-P will also be projected onto point p in the image. In other words, depth information cannot be recovered from the imaging results.

[0149] (3) Image coordinate system → pixel coordinate system: Converting physical units to pixel units. Both the pixel coordinate system and the image coordinate system lie on the imaging plane of the optical sensing node, but their origins and units of measurement differ. For example... Figure 4 As shown, the image coordinate system (x, y) generally uses the center of the image sensor as its origin, and the unit is physical units, such as mm. The pixel coordinate system (u, v) generally uses the top-left vertex of the image sensor as its origin, and the unit is pixels. The number of pixels determines the camera's resolution, as shown in... Figure 5 In the image, the camera's pixel count is M*N, meaning there are M pixels per row and N pixels per column, corresponding to a resolution of M*N. The conversion relationship between pixel coordinates [u,v] and image coordinates [x,y] is as follows: Where dx and dy represent the number of millimeters represented by each column and each row of pixels, respectively, u m and v m The coordinates of the pixel point corresponding to the center of the image sensor are equal to half of the horizontal and vertical resolutions, respectively. Represented in matrix form as follows:

[0150] The above content introduces the transformation process of four coordinate systems involved in the optical sensing imaging process. It can be understood that, combining the transformation of the second coordinate system (i.e., (2) camera coordinate system → image coordinate system) and the transformation of the third coordinate system (i.e., (3) image coordinate system → pixel coordinate system), the camera's intrinsic parameter matrix can be defined as: in, This indicates the number of pixels corresponding to the focal length in the horizontal direction. This represents the number of pixels corresponding to the focal length in the vertical direction. The horizontal field of view is known to be θ. u The relationship between focal length and field of view is as follows: Figure 6 As shown, that is Similarly, θ v This represents the field of view in the vertical direction. Therefore, when the focal length and resolution are known, or when the field of view and resolution are known, the camera's intrinsic parameter matrix can be determined.

[0151] 1.2 Radio Frequency Sensing

[0152] Radio frequency sensing (RFS) detects targets by receiving their echoes. The target being sensed (or the sensing target) can reflect, diffract, or scatter signals, etc. Figure 7 As shown, radio frequency sensing can obtain multiple 3D scattering points. It is understood that in the embodiments of this application, 3D scattering points may also be referred to as point cloud data, 3D point cloud data, or other possible names, and the embodiments of this application do not limit this.

[0153] 2. Multimodal fusion sensing

[0154] Multimodal fusion sensing can fuse radio frequency (RF) and optical multimodal information. For example, multimodal fusion sensing can determine the surface containing a target based on 3D scattering points from RF sensing; and then project the optical imaging results back onto the surface determined by RF sensing, using the intersection points to determine the target shape. Figure 7 As shown below, the implementation idea of ​​multimodal fusion sensing will be introduced in detail:

[0155] (1) Regarding optical sensing:

[0156] First, obtain the extrinsic and intrinsic parameter matrices of the camera; then, based on the intrinsic parameter matrix, project the 2D image back onto the camera coordinate system to obtain the set R of ray equations for key points (such as boundary pixels) in the image. E ={γ0*(X0,Y0,1),γ1*(X1,Y1,1),…,γ K *(X K ZK ,1)}. The specific back projection process is represented as follows:

[0157] It is understandable that, due to the unknown depth, the ray equation of the key points in the image (such as boundary pixels) is γ0*(X0,Y0,1), where (u0,v0) represents the coordinates in the pixel coordinate system, K is the camera intrinsic parameter matrix, and γ0 represents the Z-axis coordinate in the camera coordinate system, i.e., the depth information, which is the parameter to be estimated.

[0158] (2) Regarding radio frequency sensing:

[0159] Based on 3D scattering point P S ={(X0,Y0,Z0),(X1,Y1,Z1),……,(X N ,Y N Z N )}, determine the surface of the object F(x,y,z)=0.

[0160] (3) Multimodal fusion:

[0161] The ray equation R where the key point is located E Substitute the surface equation F(x,y,z)=0 to calculate the depth γ, and determine the coordinates of the key points and the shape of the object.

[0162] As we've discussed, the above content introduces multimodal fusion sensing. However, how to improve the performance of multimodal sensing is currently a hot topic of discussion.

[0163] To address the aforementioned technical problems, this application proposes the following technical solutions to improve multimodal sensing performance.

[0164] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0165] The technical solutions of this application can be applied to various communication systems, such as 4th generation (4G) mobile communication systems, such as long term evolution (LTE) systems, 5th generation (5G) mobile communication systems, such as new radio (NR) systems, and communication systems that evolve after 5G, such as future communication network systems. They can also be applied to wireless fidelity (WiFi) systems, vehicle to everything (V2X) communication systems, device-to-device (D2D) communication systems, vehicle networking communication systems, etc.

[0166] This application will present various aspects, embodiments, or features relating to systems that may include multiple devices, components, modules, etc. It should be understood and appreciated that individual systems may include additional devices, components, modules, etc., and / or may not include all the devices, components, modules, etc. discussed in conjunction with the accompanying drawings. Furthermore, combinations of these approaches are also possible.

[0167] Furthermore, in the embodiments of this application, the words "exemplary," "for example," etc., are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the term "exemplary" is intended to present the concept in a concrete manner.

[0168] In the embodiments of this application, the terms "information," "signal," "message," "channel," and "signaling" may sometimes be used interchangeably. It should be noted that, without emphasizing their distinction, their intended meanings are consistent. Similarly, "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing their distinction, their intended meanings are consistent. Furthermore, the " / " mentioned in this application can be used to indicate an "or" relationship.

[0169] The network architecture and business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0170] To facilitate understanding of the embodiments of this application, the communication system applicable to the embodiments of this application will be described first.

[0171] The communication system includes: a first device and a second device.

[0172] The first device can be used for communication and data processing. For example, the first device can receive three-dimensional point cloud data from other devices and process the three-dimensional point cloud data. The first device can also be used for radio frequency sensing of objects. The first device can be a terminal device, or a communication module, circuit with communication function, chip, chip system, or other component or assembly in a terminal device. The first device can also be a network device, or a communication module, circuit with communication function, chip, chip system, or other component or assembly in a network device.

[0173] The second device can be used for communication, for example, the first device can receive two-dimensional images from other devices. The second device can also be used for data processing, for example, the second device can process two-dimensional images. The second device can also be used for optical sensing of objects. The second device can be a terminal device, or a communication module, circuit with communication function, chip, chip system, or other component or assembly within a terminal device. The second device can also be a network device, or a communication module, circuit with communication function, chip, chip system, or other component or assembly within a network device.

[0174] The aforementioned terminal equipment can also be referred to as user equipment (UE), mobile station (MS), mobile terminal (MT), user device, access terminal, user unit, user station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent, or user device, etc., or equipment used to provide voice or data connectivity to users, and can also be Internet of Things (IoT) devices. For example, terminal equipment includes handheld devices with wireless connectivity, vehicle-mounted devices, etc. Currently, terminals can include: mobile phones, tablets, computers with wireless transceiver capabilities, laptops, handheld computers, mobile internet devices (MIDs), wearable devices (such as smartwatches, smart bracelets, pedometers, etc.), in-vehicle equipment (such as cars, bicycles, electric vehicles, airplanes, ships, trains, high-speed trains, etc.), satellite terminals, virtual reality (VR) devices, augmented reality (AR) devices, smart point-of-sale (POS) machines, customer-premises equipment (CPE), wireless terminals in industrial control, smart home devices (such as refrigerators, televisions, air conditioners, electricity meters, etc.), smart robots, robotic arms, workshop equipment, wireless terminals in autonomous driving, wireless terminals in telemedicine, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, or wireless terminals in smart homes, and flying equipment (such as smart robots, hot air balloons, drones, airplanes), etc. Terminal devices can also be other devices with terminal functions. For example, a terminal device can also be a device that plays a terminal function in device-to-device (D2D) communication.

[0175] The aforementioned network devices can be base stations, evolved NodeBs (eNodeBs), transmitting and receiving points (TRPs), transmitting points (TPs), next-generation NodeBs (gNBs), next-generation base stations in future communication systems, base stations in future mobile communication systems, satellites, or access points (APs) in WiFi systems, such as home gateways, routers, servers, switches, and bridges. They can also be integrated access and backhaul (IAB) nodes, network devices in mobile switching center non-terrestrial network (NTN) communication systems, meaning they can be deployed on high-altitude platforms or satellites. Network devices can be macro base stations, micro base stations or indoor stations, relay nodes or donor nodes, or wireless controllers in cloud-radio access network (C-RAN) scenarios. Network devices can also function as base stations in D2D communication, vehicle-to-everything (V2X) communication, drone communication, and machine-to-machine (M2M) communication. Optionally, network devices can also be servers, wearable devices, vehicles, or in-vehicle equipment. For example, the access network equipment in vehicle-to-everything (V2X) technology can be a roadside unit (RSU). Furthermore, the aforementioned network devices can also be core network elements, such as policy control functions (PCF), access and mobility management functions (AMF), location management functions (LMF), sensing management functions (SMF), or sensing functions (SF), etc.

[0176] It is understood that the terms "first device" and "second device" in the embodiments of this application are merely exemplary expressions and can be replaced with any possible expression, such as "first device" can be replaced with "radio frequency sensing node" or "first node", and "second device" can be replaced with "optical sensing node" or "second node", etc. The embodiments of this application do not limit this.

[0177] In this communication system, the first device can obtain the neighborhood matching degree of each of the N point cloud data based on the information provided by the second device, and determine the noise in the N point cloud data based on the neighborhood matching degree of each of the N point cloud data. The neighborhood matching degree of the i-th point cloud data is determined based on the image similarity between the first neighborhood i and the second neighborhood i. The first neighborhood i is the region where the i-th point cloud data is projected onto the first image, and the second neighborhood i is the region where the i-th point cloud data is projected onto the second image. i is an integer from 1 to N, and N is an integer greater than 1. It can be understood that after projecting the real point cloud data (i.e., point cloud data excluding noise) from the N point cloud data onto the first and second images, the imaging results of the similar (or identical) regions of the target object corresponding to the first and second neighborhoods of that point cloud data are considered to have a high degree of similarity; conversely, the first and second neighborhoods corresponding to the noisy point cloud data in the N point cloud data have a low degree of similarity. In this way, noise in N point cloud data can be effectively filtered out based on the neighborhood matching degree of each of the N point cloud data. That is, the point cloud data other than noise can be used for multimodal fusion perception, thereby improving the performance of multimodal perception.

[0178] In addition, the above communication system may also include other network devices and / or other terminal devices, which can be flexibly configured according to the actual situation without any restrictions.

[0179] For ease of understanding, the following will combine... Figure 1 The communication method provided in the embodiments of this application will be described in detail.

[0180] For example, Figure 7 Flowchart of the communication method provided in the embodiments of this application Figure 8 This method can be applied to the interaction between the first and second devices in the aforementioned communication system.

[0181] like Figure 8 As shown, the flow of this communication method is as follows:

[0182] S701, the first device acquires the neighborhood matching degree of each of the N point cloud data.

[0183] S702, the first device determines the noise in the N point cloud data based on the neighborhood matching degree of each of the N point cloud data.

[0184] The steps described above will be explained in detail below.

[0185] For S701:

[0186] The N point cloud data are three-dimensional point cloud data obtained by sensing the target object, or in other words, N point cloud data are point cloud data obtained by three-dimensional sensing of the target object, where N is an integer greater than 1. For example, the N point cloud data are three-dimensional point cloud data obtained by radio frequency sensing of the target object. The N point cloud data can be used to generate a stereo image of the target object. Furthermore, the N point cloud data can be data obtained by the first device through three-dimensional sensing of the target object, or data acquired by the first device from other devices; there are no restrictions.

[0187] It is understood that the term "point cloud data" in the embodiments of this application is merely an exemplary expression, and "point cloud data" can be replaced with any possible expression, such as "three-dimensional scattering points" or "three-dimensional point cloud data", etc. The embodiments of this application do not limit this.

[0188] The neighborhood matching degree of the i-th point cloud data in N point cloud data is determined based on the image similarity between the first neighborhood i and the second neighborhood i, where i is an integer from 1 to N.

[0189] The first neighborhood i is the region where the i-th point cloud data is projected (or mapped) onto the first image (described below). In other words, the first neighborhood i is the region where the point (denoted as the first projection point i) of the i-th point cloud data is projected onto the first image. That is, the first neighborhood includes the first projection point i. The first neighborhood i can be a rectangular region centered on a certain pixel (such as the first projection point i). The size of this rectangular region can be determined by the width and height of the first neighborhood i.

[0190] For example, the expression for the first neighborhood i is: Nr(u,v)={(u+x,v+y)|-r≤x≤r,-k≤y≤k}, where (u,v) are the coordinates of the point i mapped from the point cloud data to the first image, i.e., the coordinates of the first projection point i, (2r+1) is greater than or equal to 1, and (2k+1) is greater than or equal to 1; in this case, the size of the first neighborhood i is (2r+1)×(2k+1). It can be understood that | is a separator, and (u+x,v+y) before | indicates the pixel coordinates of the first neighborhood i, while the values ​​after | represent the range of x and y. The first neighborhood i can also be a circular region centered on a certain pixel (such as the first projection point i), where the pixels within this circular region are the set of all pixels whose Euclidean distance from the center pixel is less than or equal to the radius of the circle. It is understandable that when r is greater than the horizontal resolution of the first image, the first neighborhood i can be filled with zeros (or other values); when k is greater than the vertical resolution of the first image, the first neighborhood i can be filled with zeros (or other values).

[0191] For example, the expression for the first neighborhood i is: Nr(u,v)={(u+x,v+y)|x 2 +y 2 ≤r 2 Let (u, v) be the coordinates of the point i mapped from the i-th point cloud data to the first image, i.e., the coordinates of the first projection point i, where r is greater than or equal to 1. It can be understood that | is a separator; (u+x, v+y) before | indicates the pixel coordinates of the first neighborhood i, and the values ​​after | are the ranges for x and y. The first neighborhood can also be a region of other shapes and / or sizes centered on a certain pixel (such as the first projection point i), and can be flexibly set according to actual conditions without restriction. It can be understood that when r is greater than the horizontal or vertical resolution of the first image, the first neighborhood i can be filled with zeros (or other values).

[0192] The second neighborhood i is the region where the i-th point cloud data is projected (or mapped) onto the second image. In other words, the second neighborhood i is the region where the point (denoted as the second projection point i) of the i-th point cloud data is projected onto the second image. The method for determining the second neighborhood i is similar to that for determining the first neighborhood i, and can be understood by referring to the relevant introduction to the first neighborhood i above; it will not be repeated here.

[0193] The first and second images are obtained through two-dimensional perception of the target object. For example, the first and second images are obtained through optical perception of the target object. Furthermore, the first and second images are different. For instance, when the two-dimensional perception is optical, the first and second images are images obtained by perceiving (or capturing) the target object at different angles, i.e., the first and second images are captured from different angles; or, the first and second images are images obtained by perceiving the target object using two different devices located at different positions.

[0194] The following is a specific example to illustrate the above.

[0195] For example, such as Figure 8 As shown, the first image is a two-dimensional image obtained by camera C1 capturing the target object, and the i-th point cloud data ( Figure 8 Point P is projected onto the first image to obtain point P1, and the region where point P1 is located (i.e., Optical perception node capabilities The first neighborhood is the bounding box containing point P1; the second image is a two-dimensional image obtained by camera C2 capturing the target object. The i-th point cloud data is projected onto the second image to obtain point P2, and the area containing point P2 (i.e., the bounding box containing point P1) is the first neighborhood i .... State The box containing point P2 is the second neighborhood i. The first and second images were captured from different angles.

[0196] The above content describes the neighborhood matching degree of each of the N point cloud data sets. It can be understood that the first device can obtain the neighborhood matching degree of each of the N point cloud data sets, i.e., the neighborhood matching degree of the i-th point cloud data set, based on different information. The following explains the different cases.

[0197] Case 7.1: The first device determines the neighborhood matching degree of the i-th point cloud data according to the first information, which is used to indicate the relevant information of the pixels in the first neighborhood i and the second neighborhood i.

[0198] In other words, the first device acquiring the neighborhood matching degree of each of the N point cloud data can specifically include: the first device acquiring first information and determining the neighborhood matching degree of the i-th point cloud data based on the first information, wherein the first information is used to indicate the relevant information of the pixels in the first neighborhood i and the second neighborhood i. That is, the first device can determine the neighborhood matching degree of the i-th point cloud data based on the relevant information of the pixels in the first neighborhood i and the second neighborhood i, such as the pixel value corresponding to each pixel, or based on the image features determined by the pixels. Since the relevant information of the pixels can characterize the image features of the first neighborhood i and the second neighborhood i, the first device can accurately determine the neighborhood matching degree of the i-th point cloud data through the relevant information of the pixels in the first neighborhood i and the second neighborhood i.

[0199] The aforementioned first information may include first neighborhood information i and second neighborhood information i. The first neighborhood information i includes the pixel value corresponding to the pixel in the first neighborhood i. The second neighborhood information i includes the pixel value corresponding to the pixel in the second neighborhood i. That is, in this case, the relevant information about the pixel refers to the pixel value corresponding to the pixel. The first information may also include first neighborhood feature i and second neighborhood feature i. The first neighborhood feature i is used to indicate the image features of the first neighborhood i. The second neighborhood feature i is used to indicate the image features of the second neighborhood i. Image features are related to pixels; that is, the image features corresponding to a region can be determined through the pixels in the region. That is, in this case, the relevant information about the pixel refers to the image features. Of course, the first information can also be other information, and can be set according to the actual situation without limitation.

[0200] The aforementioned first information can be obtained from the second device. For example, the first device obtaining the first information may specifically include: the first device sending a first message to the second device, and correspondingly, the second device receiving the first message from the first device. The first message requests information about the pixels in the region where N point cloud data are projected onto a two-dimensional image, where the two-dimensional image is obtained by sensing a target object. In response to the first message, the second device sends first information to the first device, and correspondingly, the first device receives the first information from the second device. That is, the first device can request the second device to send information about the pixels in the region where N point cloud data are projected onto a two-dimensional image. In this case, the second device can project the i-th point cloud data onto the first image and the second image, and obtain information about the pixels in the first neighborhood i and the second neighborhood i.

[0201] The first message may include neighborhood description information, which indicates the size and / or shape of the region where the N point cloud data are projected onto the two-dimensional image. That is, after projecting the i-th point cloud data onto the first and second images, the second device can determine the first neighborhood i and the second neighborhood i based on the neighborhood description information, and then obtain relevant information about the pixels in the first and second neighborhoods i. It is understood that the first and second devices can also pre-agree or pre-define the size and / or shape of the region where the point cloud data is projected onto the two-dimensional image, without restriction.

[0202] For example, the neighborhood description information indicates that the region where the i-th point cloud data is projected onto the 2D image has a rectangular shape, and the size of this region is (2r+1)×(2k+1), where (2r+1) is greater than or equal to 1 and (2k+1) is greater than or equal to 1. After receiving the first message and projecting the i-th point cloud data onto the first image, the second device can determine the first neighborhood i based on the neighborhood description information, with the first projection point i as the center and the size of the region being (2r+1)×(2k+1). It can be understood that the specific implementation principle of the second device determining the second neighborhood i is similar to that of the second device determining the first neighborhood i, and can be understood by referring to the relevant introduction on the second device determining the first neighborhood i; it will not be repeated here.

[0203] For example, the first and second devices pre-determine that the region where the point cloud data is projected onto the two-dimensional image is circular, and the neighborhood description information is used to indicate that the size of the region where the i-th point cloud data is projected onto the two-dimensional image is πr. 2 The second device, after receiving the first message and projecting the i-th point cloud data onto the first image, can, based on the neighborhood description information, define a region centered on the first projection point i with a size of πr. 2The first neighborhood i is determined. It can be understood that the specific implementation principle of the second device determining the second neighborhood i is similar to that of the second device determining the first neighborhood i. For further understanding, please refer to the relevant introduction on the second device determining the first neighborhood i; it will not be elaborated here.

[0204] The first message can also include N point cloud data points, that is, the coordinates corresponding to the N point cloud data points. It can be understood that the N point cloud data points can be sent to the second device through other messages, such as the first device sending the first message and then sending the N point cloud data points to the second device. In other words, the N point cloud data points and the neighborhood description information do not necessarily need to be carried in the same message; this can be flexibly set according to the actual situation without any restrictions.

[0205] It is understandable that the first device can request information about the pixels in the region where N point cloud data are projected onto the two-dimensional image, based on the reporting capabilities of the second device.

[0206] In a first possible implementation, when the first information includes first neighborhood information i and second neighborhood information i, and before the first device sends the first message to the second device, the communication method may further include: the second device sending a first capability parameter to the first device, and correspondingly, the first device receiving the first capability parameter from the second device, the first capability parameter being used to indicate that the second device supports providing the pixel values ​​corresponding to pixels in a first region to the first device, the first region being the region where the point cloud data is projected onto the two-dimensional image; the first device sending the first message to the second device may specifically include: the first device sending the first message to the second device according to the first capability parameter.

[0207] The first capability parameter can also be understood as the second device's ability to provide the first device with the pixel values ​​corresponding to the pixels in the first region, or the second device having the capability to provide the first device with the pixel values ​​corresponding to the pixels in the first region. It can be understood that when the second device is a chip, the device containing the second device has the capability to provide the first device with the pixel values ​​corresponding to the pixels in the first region.

[0208] The first device sending a first message to the second device based on the first capability parameter can be understood as the first device determining to send a first message to the second device based on the first capability parameter.

[0209] In this embodiment, after receiving the first capability parameter, the first device can determine that the second device (or the device containing the second device) has the capability to provide the first device with the pixel values ​​corresponding to the pixels in the first region. At this time, the first device can send a first message to the second device, requesting the second device to send relevant information about the region where N point cloud data are projected onto the 2D image. This avoids the first device failing to request the second device to send relevant information about the region where N point cloud data are projected onto the 2D image, thus preventing additional communication overhead.

[0210] Furthermore, the aforementioned first message, used to request information about pixels in the region where the N point cloud data are projected onto the 2D image, can specifically include: the first message requesting pixel values ​​corresponding to the pixels in the region where the N point cloud data are projected onto the 2D image. This avoids the second device providing incorrect information to the first device, such as providing image features corresponding to the region where the N point cloud data are projected onto the 2D image, thus preventing additional communication overhead.

[0211] Furthermore, the first capability parameter can also be used to indicate at least one of the following information of the second device: information protection level information, sensing capability information, communication capability information, computing capability information, storage capability information, internal and external parameter transmission information, neighborhood feature acquisition information, or neighborhood matching degree acquisition information. Each piece of information is described in detail below.

[0212] Information protection level information is used to indicate the degree of information exposure supported by the second device, such as whether the second device is willing to send information such as the internal and external parameters of the second device, the relevant information of the pixels in the area where the point cloud data is projected onto the two-dimensional image, to other devices (such as the first device).

[0213] The perception capability information is used to indicate the perception-related parameters of the second device, such as the resolution and field of view of the second device.

[0214] Communication capability information describes whether the second device has the ability to share data, and the strength of that ability. The ability to share data is related to factors such as bandwidth and signal-to-noise ratio, which affect the information transmitted by the second device. For example, can the second device transmit large amounts of two-dimensional images, or can the second device transmit information about the pixels in the region where point cloud data is projected onto the two-dimensional image?

[0215] Computational capability information indicates whether the second device has data analysis and computation capabilities, and the strength of these capabilities. These capabilities are related to the configuration of the second device's central processing unit (CPU), graphics processing unit (GPU), and other units. These configurations affect the data ultimately transmitted by the second device, such as whether it transmits low-computational-requirement 2D images or high-computational-requirement point cloud data projected onto the 2D image and containing information about the pixels in that region.

[0216] Storage capacity information indicates the storage capacity of the second device. This storage capacity information is related to the configuration of the second device, such as its memory and disk. The storage capacity of the second device affects its participation in multimodal sensing. For example, if the second device caches a large amount of sensing data, the sensing cost required is reduced compared to the second device re-sensing and collecting sensing data.

[0217] The internal and external parameter transmission information is used to indicate whether the second device supports the transmission of internal and external parameters (i.e., internal parameters and external parameters) and two-dimensional images. The internal parameter can be at least one of the following parameters of the second device: focal length, resolution, or field of view. The external parameter can be at least one of the following parameters of the second device: position or orientation. The two-dimensional image is an image obtained by the second device through two-dimensional perception of the object (such as optical perception).

[0218] Neighborhood feature acquisition information can also be called neighborhood feature extraction and transmission information. This neighborhood feature acquisition information is used to indicate whether the second device supports neighborhood feature extraction and the transmission of the extracted neighborhood features. The neighborhood is the region where the point cloud data is projected onto the two-dimensional image. For details, please refer to the aforementioned introductions of "first neighborhood i" and "second neighborhood i" for further understanding, which will not be repeated here. Neighborhood features can be understood as the image features corresponding to the neighborhood. These image features are related to pixels, such as feature extraction based on the pixel values ​​corresponding to the pixels in the neighborhood. It can be understood that sending neighborhood features to other devices by the second device, compared to sending neighborhood information to other devices (see the related introduction of neighborhood extraction and acquisition information below), can further protect privacy and reduce data transmission volume.

[0219] The neighborhood matching degree acquisition information can also be called the matching degree calculation and transmission information. This neighborhood matching degree acquisition information is used to indicate whether the second device supports providing neighborhood matching degrees to other devices (such as the first device). That is, the second device can receive data (such as point cloud data) from other devices, and calculate and transmit the neighborhood matching degree based on this data.

[0220] The content indicated by the above information can be flexibly set according to the actual situation without restriction. Furthermore, the names of the above information are merely exemplary; the specific names can be flexibly set according to the actual situation without restriction.

[0221] For example, the first capability parameter is also used to indicate internal and external parameter transmission information, neighborhood feature acquisition information, and neighborhood matching degree acquisition information. The internal and external parameter transmission information indicates that the second device does not support transmitting internal and external parameters and two-dimensional images to other devices. The neighborhood feature acquisition information indicates that the second device supports neighborhood feature extraction and transmission of the extracted neighborhood features. The neighborhood matching degree acquisition information indicates that the second device does not support providing neighborhood matching degrees to other devices.

[0222] Continuing the example above, when the second device reports its capabilities to the first device, if the second device and the first device have pre-agreed on the capabilities that need to be reported, the second device can indicate the status of each capability corresponding to itself to the first device based on the pre-agreed capabilities. That is, the first capability parameter is the parameter indicating the status of each capability corresponding to the second device. For example, as shown in Table 1 below, the first device and the second device have pre-agreed on the capabilities of the optical sensing node to be reported, namely, internal and external parameter transmission information, neighborhood extraction information (described below), neighborhood feature acquisition information, and neighborhood matching degree acquisition information. The second device can send the status corresponding to each capability to the first device based on the capabilities of the optical sensing node and its own situation.

[0223] Table 1

[0224] Internal and external parameter transmission capabilities No Neighborhood extraction acquisition capabilities Yes Neighborhood feature acquisition capabilities Yes Neighborhood matching degree acquisition capabilities No Figure 9 Figure 9

[0225] It is understood that in this embodiment of the application (i.e., the first possible implementation), the state of the neighborhood extraction capability is "yes". The states of other capabilities in Table 1 besides the neighborhood extraction capability can be set according to the actual situation without restriction.

[0226] In this embodiment of the application, by indicating the capabilities of the second device to the first device through the first capability parameter, the first device can determine the capabilities of the second device (or the device in which the second device is located) based on the first capability parameter, and thus determine the operation of the second device in multimodal sensing based on the capabilities of the second device (or the device in which the second device is located).

[0227] Furthermore, when the first information includes first neighborhood information i and second neighborhood information i, the determination of the neighborhood matching degree of the i-th point cloud data by the first device based on the first information can specifically include: the first device determining the neighborhood matching degree of the i-th point cloud data based on the first neighborhood information i, the second neighborhood information i, and the matching degree calculation method, wherein the matching degree calculation method is any one of the following: normalized squared difference, cumulative density function, or neighborhood feature calculation (described below). It is understood that the matching degree calculation method can also be other methods, without limitation. In this way, the first device can accurately determine the neighborhood matching degree of the i-th point cloud data.

[0228] In a second possible implementation, where the first information includes a first neighborhood feature i and a second neighborhood feature i, and before the first device sends the first message to the second device, the communication method may further include: the second device sending a second capability parameter to the first device, and correspondingly, the first device receiving the second capability parameter from the second device, the second capability parameter indicating that the second device supports providing neighborhood features to the first device, the neighborhood features indicating the image features of the region where the point cloud data is projected onto the two-dimensional image; the first device sending the first message to the second device may specifically include: the first device sending the first message to the second device according to the second capability parameter; or, the first device determining to send the first message to the second device according to the second capability parameter.

[0229] The second capability parameter can also be understood as the second device's ability to provide neighborhood features to the first device, or the second device having the capability to provide neighborhood features to the first device. It can be understood that when the second device is a chip, the device containing the second device has the capability to provide neighborhood features to the first device.

[0230] In this embodiment, after receiving the second capability parameter, the first device can determine that the second device (or the device containing the second device) has the capability to provide neighborhood features to the first device. At this time, the first device can (determine) send a first message to the second device, requesting the second device to send image features corresponding to the region where N point cloud data are projected onto the 2D image. This avoids the first device failing to request the second device to send image features corresponding to the region where N point cloud data are projected onto the 2D image, thus preventing additional communication overhead.

[0231] Furthermore, the aforementioned first message, used to request information about pixels in the region where the N point cloud data are projected onto the 2D image, can specifically include: the first message requesting image features corresponding to the region where the N point cloud data are projected onto the 2D image. This avoids the second device providing incorrect information to the first device, such as providing pixel values ​​corresponding to pixels in the region where the N point cloud data are projected onto the 2D image, thus preventing additional communication overhead.

[0232] Furthermore, the second capability parameter is also used to indicate at least one of the following information of the second device: information protection level information, sensing capability information, communication capability information, computing capability information, storage capability information, internal and external parameter transmission information, neighborhood extraction acquisition information, or neighborhood matching degree acquisition information.

[0233] The neighborhood extraction information can also be called neighborhood extraction and transmission information, or other names, without limitation. This neighborhood extraction information is used to indicate whether the second device supports neighborhood extraction and to transmit the extracted neighborhood information. The neighborhood is the region where the point cloud data is projected onto the two-dimensional image. For details, please refer to the aforementioned introductions of "first neighborhood i" and "second neighborhood i" for further understanding, which will not be repeated here. Neighborhood extraction can be understood as obtaining relevant information about the neighborhood, such as the pixel values ​​corresponding to the pixels in the neighborhood.

[0234] For other information, please refer to the aforementioned related introductions, which will not be repeated here.

[0235] For example, the second capability parameter is also used to indicate internal and external parameter transmission information, neighborhood extraction acquisition information, and neighborhood matching degree acquisition information. The internal and external parameter transmission information indicates that the second device does not support transmitting internal and external parameters and two-dimensional images to other devices. The neighborhood extraction acquisition information indicates that the second device does not support neighborhood extraction and transmission operations. The neighborhood matching degree acquisition information indicates that the second device does not support providing neighborhood matching degrees to other devices.

[0236] Continuing the example above, when the second device reports its capabilities to the first device, if the second device and the first device have pre-agreed on the capabilities that need to be reported, the second device can indicate the status of each capability corresponding to the second device to the first device according to the pre-agreed capabilities. For details, please refer to the aforementioned related descriptions; they will not be repeated here. It is understood that in this embodiment of the application (i.e., the second possible implementation), the status of the neighborhood feature acquisition capability is "yes," and the status of the optical sensing node capabilities other than the neighborhood feature acquisition capability can be set according to the actual situation without restriction.

[0237] In this embodiment of the application, by indicating the capabilities of the second device to the first device through the second capability parameters, the first device can determine the capabilities of the second device based on the second capability parameters, and thus determine the operation of the second device in multimodal sensing based on the capabilities of the second device.

[0238] Furthermore, when the first information includes the first neighborhood feature i and the second neighborhood feature i, the first device determining the neighborhood matching degree of the i-th point cloud data based on the first information can specifically include: the first device determining the neighborhood matching degree of the i-th point cloud data based on the first neighborhood feature i, the second neighborhood feature i, and the neighborhood feature calculation method (described below). In this way, the first device can accurately determine the neighborhood matching degree of the i-th point cloud data.

[0239] Case 7.2: The first device obtains the neighborhood matching degree of the i-th point cloud data from the second device.

[0240] That is, the first device acquiring the neighborhood matching degree of each of the N point cloud data can specifically include: the first device sending a second message to the second device, and correspondingly, the second device receiving the second message from the first device. The second message is used to request the neighborhood matching degree corresponding to the N point cloud data. The neighborhood matching degree corresponding to the N point cloud data is used to indicate the image similarity of the region where the i-th point cloud data is projected onto multiple two-dimensional images. The multiple two-dimensional images are obtained by perceiving the target object. In response to the second message, the second device sends the neighborhood matching degree of each of the N point cloud data to the first device, and correspondingly, the first device receives the neighborhood matching degree of each of the N point cloud data from the second device.

[0241] The second message mentioned above may include neighborhood description information, which can be referred to in the relevant introduction in "Case 7.1" above, and will not be repeated here.

[0242] The second message can also include N point cloud data points, that is, the coordinates corresponding to the N point cloud data points. It can be understood that the N point cloud data points can be sent to the second device through other messages, such as the first device sending the second message and then sending the N point cloud data points to the second device. In other words, the N point cloud data points and the neighborhood description information do not necessarily need to be carried in the same message; this can be flexibly set according to the actual situation without any restrictions.

[0243] In this embodiment, the first device can request neighborhood matching degrees corresponding to N point cloud data from the second device. After receiving the request, the second device can calculate the neighborhood matching degrees corresponding to the N point cloud data based on the N point cloud data and multiple two-dimensional images, and send the calculated neighborhood matching degrees of each of the N point cloud data to the first device. This reduces the computational overhead of the first device.

[0244] It is understandable that the first device can request the second device to send the neighborhood matching degree of each of the N point cloud data based on the reporting capability of the second device.

[0245] For example, before the first device sends the second message to the second device, the communication method may further include: the second device sending a third capability parameter to the first device, and correspondingly, the first device receiving the third capability parameter from the second device. The third capability parameter is used to indicate that the second device supports providing the first device with the neighborhood matching degree corresponding to the three-dimensional point cloud data. The neighborhood matching degree corresponding to the three-dimensional point cloud data is used to indicate the image similarity of the region where the three-dimensional point cloud data is projected onto multiple two-dimensional images. The first device sending the second message to the second device may specifically include: the first device sending the second message to the second device according to the third capability parameter.

[0246] The third capability parameter can also be understood as the second device's ability to provide the first device with the neighborhood matching degree corresponding to the 3D point cloud data, or the second device's ability to provide the first device with the neighborhood matching degree corresponding to the 3D point cloud data. It can be understood that when the second device is a chip, the device containing the second device has the ability to provide the first device with the neighborhood matching degree corresponding to the 3D point cloud data.

[0247] The first statement that the device sends a second message to the second device based on the third capability parameter can be understood as the first device determining to send a second message to the second device based on the third capability parameter.

[0248] In this embodiment, after receiving the third capability parameter, the first device can determine that the second device (or the device containing the second device) has the capability to provide the first device with neighborhood matching degrees corresponding to the 3D point cloud data. At this time, the first device can (determine) send a second message to the second device, requesting the second device to send neighborhood matching degrees corresponding to N point cloud data. This avoids the first device failing to request the second device to send neighborhood matching degrees corresponding to N point cloud data, thus preventing additional communication overhead.

[0249] Furthermore, the third capability parameter is also used to indicate at least one of the following information of the second device: information protection level information, sensing capability information, communication capability information, computing capability information, storage capability information, internal and external parameter transmission information, neighborhood extraction and acquisition information, or neighborhood feature acquisition information. Each piece of information can be referred to the relevant description in “Situation 7.1” above, and will not be repeated here.

[0250] For example, the third capability parameter is also used to indicate internal and external parameter transmission information, neighborhood extraction acquisition information, and neighborhood feature acquisition information. The internal and external parameter transmission information indicates that the second device does not support transmitting internal and external parameters and two-dimensional images to other devices. The neighborhood extraction acquisition information indicates that the second device does not support neighborhood extraction and transmits the extracted neighborhood information. The neighborhood feature acquisition information indicates that the second device does not support neighborhood feature extraction but supports transmitting the extracted neighborhood features.

[0251] Continuing the example above, when the second device reports its capabilities to the first device, if the second device and the first device have pre-agreed on the capabilities that need to be reported, the second device can indicate the status of each capability to the first device according to the pre-agreed capabilities. For details, please refer to the relevant description in "Situation 7.1" above; it will not be repeated here. It can be understood that in Situation 7.2, the status of the neighborhood matching degree acquisition capability is "yes," and the status of the optical sensing node capabilities other than the neighborhood matching degree acquisition capability can be set according to the actual situation without restriction.

[0252] In this embodiment of the application, by indicating the capabilities of the second device to the first device through the third capability parameter, the first device can determine the capabilities of the second device based on the third capability parameter, and thus determine the operation of the second device in multimodal sensing based on the capabilities of the second device.

[0253] Optionally, the above communication method may further include: a second device projecting the i-th point cloud data onto a first image and a second image; the second device determining a first neighborhood i and a second neighborhood i based on the first image and the second image mapped with the i-th point cloud data; the second device determining first neighborhood information i and second neighborhood information i based on the first neighborhood i and the second neighborhood i, wherein the first neighborhood information i includes the pixel value corresponding to the pixel in the first neighborhood i, and the second neighborhood information i includes the pixel value corresponding to the pixel in the second neighborhood i; and the second device determining the neighborhood matching degree of the i-th point cloud data based on the first neighborhood information i and the second neighborhood information i (described below). In this way, the second device can accurately obtain the neighborhood matching degree of the i-th point cloud data.

[0254] Optionally, the above communication method may further include: a second device projecting the i-th point cloud data onto a first image and a second image; the second device determining a first neighborhood i and a second neighborhood i based on the first image and the second image mapped with the i-th point cloud data; the second device determining a first neighborhood feature i and a second neighborhood feature i based on the first neighborhood i and the second neighborhood i, where the first neighborhood feature i indicates the image features of the first neighborhood i and the second neighborhood feature i indicates the image features of the second neighborhood i; and the second device determining the neighborhood matching degree of the i-th point cloud data based on the first neighborhood feature i and the second neighborhood feature i (described below). In this way, the second device can accurately obtain the neighborhood matching degree of the i-th point cloud data.

[0255] Case 7.3: The first device determines the neighborhood matching degree of the i-th point cloud data based on the parameter information, the first image, and the second image. The parameter information is used to project N point cloud data onto the first image and the second image.

[0256] In other words, the first device's acquisition of the neighborhood matching degree of each of the N point cloud data points can specifically include: the first device acquiring parameter information, a first image, and a second image, and determining the neighborhood matching degree of each of the N point cloud data points based on the parameter information, the first image, and the second image. The parameter information is used to project the N point cloud data points onto the first image and the second image. That is, the first device can project the N point cloud data points onto the first image and the second image based on the parameter information, the first image, and the second image, and then determine the neighborhood matching degree of each of the N point cloud data points based on the projected first image and the second image. This enables the first device to accurately acquire the neighborhood matching degree of each of the N point cloud data points.

[0257] Furthermore, the acquisition of parameter information, first image, and second image by the first device may specifically include: the second device acquiring parameter information, first image, and second image, wherein the parameter information is used to indicate relevant parameters for the second device to perform two-dimensional perception of the target object, and the first image and second image are obtained by the second device to perform two-dimensional perception of the target object, and the first image and second image are different; the second device sends parameter information, first image, and second image to the first device, and correspondingly, the first device receives parameter information, first image, and second image from the second device.

[0258] The aforementioned relevant parameters can be internal and external parameters of the second device (i.e., internal and external parameters). These relevant parameters can be used to project N point cloud data onto the first and second images. For details, please refer to the conversion process from the world coordinate system to the pixel coordinate system in "1.1 Optical Perception" above; it will not be elaborated upon here.

[0259] For example, the aforementioned parameters may include at least one of the following information about the second device: focal length, resolution, field of view, position, or pointing. The focal length, resolution, and field of view can be referred to the relevant descriptions in "1.1 Optical Sensing" above, and will not be repeated here. Position refers to the location of the second device in the world, or the relative position of the second device to a certain device (such as the first device); pointing refers to the direction corresponding to the second device, such as the pointing of the second device relative to a certain device (such as the first device). The position and / or pointing of the second device can be flexibly set according to the actual situation and are not limited.

[0260] In this embodiment, the first device can obtain parameter information, a first image, and a second image from the second device. It is understood that the second device can periodically send to the first device relevant parameters for two-dimensional perception of the target object, as well as two-dimensional images (such as the first and second images) obtained from the perception of the target object. The first device can also request relevant parameters for two-dimensional perception of the target object, as well as two-dimensional images obtained from the perception of the target object, from the second device to obtain parameter information, the first image, and the second image.

[0261] For example, before the first device receives the parameter information, the first image, and the second image from the second device, the communication method may further include: the first device sending a third message to the second device, and correspondingly, the second device receiving the third message from the first device. The third message requests relevant parameters and at least one image from the second device for two-dimensional perception of the target object. Specifically, the second device sending the parameter information, the first image, and the second image to the first device may include: in response to the third message, the second device sending the parameter information, the first image, and the second image to the first device. This enables the first device to obtain the parameter information, the first image, and the second image in a timely manner.

[0262] It is understood that the first device can request the second device to send relevant parameters and at least one image of the target object for two-dimensional perception, based on the reporting capabilities of the second device.

[0263] For example, before the first device receives parameter information, the first image, and the second image from the second device, the above communication method may further include: the second device sending a fourth capability parameter to the first device, and correspondingly, the first device receiving the fourth capability parameter from the second device, the fourth capability parameter being used to indicate that the second device supports providing the first device with relevant parameters and images for two-dimensional perception of objects; the first device sending a third message to the second device may specifically include: the first device sending a third message to the second device according to the fourth capability parameter.

[0264] The fourth capability parameter can also be understood as the second device being able to provide the first device with relevant parameters and images for two-dimensional object perception, or the second device having the capability to provide the first device with relevant parameters and images for two-dimensional object perception. It can be understood that when the second device is a chip, the device containing the second device has the capability to provide the first device with relevant parameters and images for two-dimensional object perception.

[0265] The first device sending a third message to the second device based on the fourth capability parameter can be understood as the first device determining to send a third message to the second device based on the fourth capability parameter.

[0266] In this embodiment, after receiving the fourth capability parameter, the first device can determine that the second device (or the device containing the second device) has the capability to provide the first device with relevant parameters and images for two-dimensional object perception. At this time, the first device can (determine) send a third message to the second device, requesting the second device to provide relevant parameters and images for two-dimensional object perception. This avoids the first device failing to request relevant parameters and images for two-dimensional object perception from the second device, thus preventing additional communication overhead.

[0267] Furthermore, the fourth capability parameter is also used to indicate at least one of the following information of the second device: information protection level information, sensing capability information, communication capability information, computing capability information, storage capability information, neighborhood extraction acquisition information, neighborhood feature acquisition information, or neighborhood matching degree acquisition information. Each piece of information can be referred to the relevant description in the aforementioned "Situation 7.1", and will not be repeated here.

[0268] For example, the fourth capability parameter is also used to indicate neighborhood extraction information, neighborhood feature acquisition information, and neighborhood matching degree acquisition information. The neighborhood extraction information indicates that the second device supports neighborhood extraction and transmits the extracted neighborhood information. The neighborhood feature acquisition information indicates that the second device supports neighborhood feature extraction and supports transmitting the extracted neighborhood features. The neighborhood matching degree acquisition information indicates that the second device does not support providing neighborhood matching degrees to other devices.

[0269] Continuing the example above, when the second device reports its capabilities to the first device, if the second device and the first device have pre-agreed on the capabilities that need to be reported, the second device can indicate the status of each capability to the first device according to the pre-agreed capabilities. For details, please refer to the relevant description in "Situation 7.1" above; it will not be repeated here. It can be understood that in Situation 7.3, the status of the intrinsic and extrinsic parameter transmission capability is "yes," and the status of the optical sensing node capabilities other than the intrinsic and extrinsic parameter transmission capabilities can be set according to the actual situation without restriction.

[0270] In this embodiment of the application, by instructing the first device on the capabilities of the second device through the fourth capability parameter, the first device can determine the capabilities of the second device (or the device in which the second device is located) based on the third capability parameter, and thus determine the operation of the second device in multimodal sensing based on the capabilities of the second device (or the device in which the second device is located).

[0271] As can be understood, the above content describes how the first device acquires parameter information, the first image, and the second image. After acquiring the parameter information, the first image, and the second image, the first device can determine the neighborhood matching degree of the i-th point cloud data based on the parameter information, the first image, and the second image. These will be described in detail below.

[0272] In one possible implementation, the first device obtaining the neighborhood matching degree of each of the N point cloud data points based on parameter information, a first image, and a second image can specifically include: the first device projecting the i-th point cloud data point onto the first image and the second image based on the parameter information; the first device determining a first neighborhood i and a second neighborhood i based on the first image and the second image mapped with the i-th point cloud data point; the first device determining first neighborhood information i and second neighborhood information i based on the first neighborhood i and the second neighborhood i, where the first neighborhood information i includes the pixel value corresponding to the pixel point in the first neighborhood i, and the second neighborhood information i includes the pixel value corresponding to the pixel point in the second neighborhood i; and the first device determining the neighborhood matching degree of the i-th point cloud data point based on the first neighborhood information i and the second neighborhood information i (described below). In this way, the first device can accurately determine the neighborhood matching degree of the i-th point cloud data point.

[0273] In another possible implementation, the first device obtaining the neighborhood matching degree of each of the N point cloud data based on parameter information, a first image, and a second image can specifically include: the first device projecting the i-th point cloud data onto the first image and the second image based on the parameter information; the first device determining a first neighborhood i and a second neighborhood i based on the first image and the second image mapped with the i-th point cloud data; the first device determining a first neighborhood feature i and a second neighborhood feature i based on the first neighborhood i and the second neighborhood i, where the first neighborhood feature i indicates the image features of the first neighborhood i and the second neighborhood feature i indicates the image features of the second neighborhood i; and the first device determining the neighborhood matching degree of the i-th point cloud data based on the first neighborhood feature i and the second neighborhood feature i (described below). In this way, the first device can accurately determine the neighborhood matching degree of the i-th point cloud data.

[0274] It is understood that, as described above through Situations 7.1-7.3, the first device can obtain the neighborhood matching degree of each of the N point cloud data based on different information. It is also understood that, before the first device obtains the neighborhood matching degree of each of the N point cloud data (i.e., S701), the second device can report its capability parameters to the first device, so that the first device can determine the information obtained from the second device based on these capability parameters.

[0275] For example, the above communication method may further include: the second device sending a fifth capability parameter to the first device, and correspondingly, the first device receiving the fifth capability parameter from the second device. The fifth capability parameter is used to indicate at least one of the following information of the second device: information protection level information, sensing capability information, communication capability information, computing capability information, storage capability information, intrinsic and extrinsic parameter transmission information, neighborhood extraction acquisition information, neighborhood feature acquisition information, or neighborhood matching degree acquisition information. Each piece of information can be referred to the relevant description in "Situation 7.1" above, and will not be repeated here.

[0276] After receiving the fifth capability parameter, the first device can determine the information to be requested from the second device based on the fifth capability parameter.

[0277] For example, the fifth capability parameter includes internal and external parameter transmission information, which is used to instruct the second device to support the transmission of internal and external parameters and two-dimensional images to other devices. Based on this fifth capability parameter, the first device can request relevant parameters and at least one image from the second device for two-dimensional perception of the target object. Furthermore, after receiving the parameter information, the first image, and the second image from the second device, the first device can obtain the neighborhood matching degree of each of the N point cloud data points based on these parameters, images, and images.

[0278] For example, the fifth capability parameter includes internal and external parameter transmission information, neighborhood extraction acquisition information, neighborhood feature acquisition information, and neighborhood matching degree acquisition information. The internal and external parameter transmission information indicates that the second device does not support transmitting internal and external parameters and 2D images to other devices. The neighborhood extraction acquisition information indicates that the second device supports neighborhood extraction and transmits the extracted neighborhood information. The neighborhood feature acquisition information indicates that the second device supports neighborhood feature extraction and transmits the extracted neighborhood features. The neighborhood matching degree acquisition information indicates that the second device does not support providing neighborhood matching degrees to other devices. The first device can determine whether to request neighborhood information or neighborhood features from the second device based on the fifth capability parameter and the method by which the first device acquires the neighborhood matching degree of the point cloud data. If the first device calculates the neighborhood matching degree of the point cloud data using neighborhood information, then the first device can request the pixel values ​​corresponding to the pixels in the region where N point cloud data points are projected onto the 2D image from the second device. Furthermore, after receiving the neighborhood information, the first device can acquire the neighborhood matching degree of each of the N point cloud data points based on the neighborhood information.

[0279] It is understandable that if the second device does not support information exposure—that is, if the second device does not support the transmission of internal and external parameters and 2D images, does not support neighborhood extraction and transmission of extracted neighborhood information, does not support neighborhood feature extraction and transmission of extracted neighborhood features, and does not support providing neighborhood matching degrees to other devices—then it corresponds to the first device's request failing. In this case, the first device cannot obtain various information (such as internal and external parameter information, 2D images, extracted neighborhood information, etc.) from the second device. Furthermore, if the first device requests the second device to report its corresponding capability parameters, and the second device does not support information exposure, the second device may choose not to send a capability reporting response message, or it may set all the states in Table 1 above to "No" before reporting, or it may directly send a "fail" command.

[0280] The following example uses one point cloud dataset and two 2D images to illustrate a specific method for obtaining the neighborhood matching degree corresponding to the point cloud dataset. It can be understood that this method can be used to obtain the neighborhood matching degree corresponding to the aforementioned N point cloud datasets; that is, this method can be used to obtain the neighborhood matching degree of each of the N point cloud datasets.

[0281] Step 1: Project the point cloud data onto images #1 and #2 to obtain projection point K1 on image #1 and projection point K2 on image #2.

[0282] That is, projecting the point cloud data from the world coordinate system to the pixel coordinate system, or in other words, transforming the coordinates of the point cloud data to the corresponding coordinates in the pixel coordinate system.

[0283] The point cloud data is projected onto the 2D image obtained by optical sensing. The projection process is shown in the following formula:

[0284]

[0285] Among them, [X P ,Y P Z P [u] represents the coordinates of the point cloud data. p ,v p [ ] represents the coordinates of the point cloud data projected onto the pixel coordinate system. Let be the extrinsic parameter matrix of the camera. This is the intrinsic parameter matrix for transforming the camera coordinate system to the pixel coordinate system.

[0286] It is understandable that the coordinate transformation process of point cloud data can be understood by referring to the transformation process from the world coordinate system to the pixel coordinate system in "1.1 Optical Perception" above, and will not be repeated here. It is also understandable that the coordinates after projecting the point cloud data onto images #1 and #2 can be determined by the above formula (1).

[0287] Step 2: Perform neighborhood extraction on projection points K1 and K2 to obtain neighborhood #1 corresponding to projection point K1 and neighborhood #2 corresponding to projection point K2.

[0288] For example, a neighborhood is taken near the projection point K1 to obtain neighborhood #1; a neighborhood is taken near the projection point K2 to obtain neighborhood #2. The method for determining neighborhood #1 and neighborhood #2 can be understood by referring to the relevant introduction in the "first neighborhood" section above, and will not be repeated here.

[0289] It is understandable that both neighborhood #1 and neighborhood #2 include multiple pixels (u,v).

[0290] Step 3: Determine the neighborhood matching degree of the point cloud data based on neighborhood #1 and neighborhood #2.

[0291] In one possible implementation, the neighborhood matching degree is determined based on the pixel values ​​corresponding to the pixels in neighborhood #1 and the pixels corresponding to the pixels in neighborhood #2.

[0292] For example, neighborhood matching degree can be determined by normalized sum of squared difference (NSSD). The expression for calculating neighborhood matching degree using NSSD is as follows: Among them, A u,v B represents the pixel value corresponding to pixel (u,v) in neighborhood #1. u,v The pixel value corresponding to pixel (u,v) in neighborhood #2.

[0293] For example, by comparing the distribution differences of pixel values ​​in neighborhood #1 and neighborhood #2 using the cumulative distribution function (CDF), the expression for calculating the neighborhood matching degree using CDF is as follows: Where F is the density function, expressed as: h(x) represents the frequency distribution of pixel values, h(x) = ∑ u,v δ(A(u,v)-x), where δ is the Dirac delta function and x is the pixel value.

[0294] It is understandable that when calculating the neighborhood matching degree using the two methods mentioned above, the closer the neighborhood matching degree result is to 1, the higher the neighborhood matching degree is.

[0295] In another possible implementation, feature extraction is performed on the pixel values ​​corresponding to pixels in neighborhood #1 and neighborhood #2 to obtain neighborhood features #1 and neighborhood features #2. Based on neighborhood features #1 and neighborhood features #2, the neighborhood matching degree is determined. It can be understood that in this case, the neighborhood feature calculation function should satisfy the following condition: the input is the neighborhood pixel value, and the output is the feature value. The input and output need to satisfy: the higher the matching degree of the input neighborhood elements, the closer the calculated feature value is.

[0296] For example, neighborhood matching is determined through difference hashing, which involves calculating the difference between adjacent pixels row by row. Specifically, if the pixel value to the left of a neighborhood (neighborhood #1 or neighborhood #2) is greater than the pixel value to the right, it is recorded as 1; otherwise, it is recorded as 0. All the difference results in the neighborhood (neighborhood #1 or neighborhood #2) are concatenated to form a binary hash string, i.e., H. i =b i, 0b i,1 ...b i,n-1 , where b i,jThis represents the j-th hash value of the i-th neighborhood. Next, the Hamming distance between the hash strings corresponding to neighborhood #1 and neighborhood #2 is compared. Where f is an indicator function, when b i,j -b i-1,j If the distance is not equal to 0, the f function value is 1; otherwise, it is 0. It can be understood that the smaller the Hamming distance, the higher the image matching degree.

[0297] It is understandable that steps 1-3 above use two 2D images as an example to determine the neighborhood matching degree of point cloud data. When there are more than two 2D images, the point cloud data can be projected onto these multiple 2D images to obtain multiple projection points; neighborhood extraction is performed on the multiple projection points to obtain multiple neighborhoods; finally, the neighborhood matching degree of the point cloud data is determined based on the multiple neighborhoods. When determining the neighborhood matching degree of point cloud data based on multiple neighborhoods, the neighborhood matching degree corresponding to each pair of neighborhoods can be calculated, and then the average value can be calculated based on the multiple neighborhood matching degrees to obtain the final neighborhood matching degree; or, the neighborhood matching degree corresponding to each pair of neighborhoods can be calculated, and the neighborhood matching degree with the largest or smallest value among the multiple neighborhood matching degrees can be used as the final neighborhood matching degree.

[0298] For example, when there are three 2D images, the point cloud data can be projected onto the three 2D images to obtain projected points #1a, #2a, and #3a. Neighborhood extraction is performed on projected points #1a, #2a, and #3a to obtain the neighborhoods #1a, #2a, and #3a corresponding to projected points #1a, #2a, and #3a, respectively. The neighborhood matching degree #1a is calculated based on neighborhoods #1a and #2a; the neighborhood matching degree #2a is calculated based on neighborhoods #1a and #3a; and the neighborhood matching degree #3a is calculated based on neighborhoods #2a and #3a. The average of the neighborhood matching degrees #1a, #2a, and #3a is calculated to obtain the final neighborhood matching degree.

[0299] For example, when there are four 2D images, the point cloud data can be projected onto the four 2D images to obtain projection points #1b, #2b, #3b, and #4b. Neighborhood extraction is then performed on projection points #1b, #2b, #3b, and #4b to obtain the neighborhood #1b of projection point #1b, the neighborhood #2b of projection point #2b, the neighborhood #3b of projection point #3b, and the neighborhood #4b of projection point #4b. Based on neighbors #1b and #2b, the neighborhood matching degree #1b is calculated using NSSD; based on neighbors #1b and #3b, the neighborhood matching degree #2b is calculated using NSSD; based on neighbors #1b and #4b, the neighborhood matching degree #3b is calculated using NSSD; based on neighbors #2b and #3b, the neighborhood matching degree #4b is calculated using NSSD; based on neighbors #2b and #4b, the neighborhood matching degree #5b is calculated using NSSD; based on neighbors #3b and #4b, the neighborhood matching degree #6b is calculated using NSSD. The neighborhood matching degree #6b has the smallest value, and is therefore taken as the final neighborhood matching degree.

[0300] It can also be understood that the neighborhood matching degree of the i-th point cloud data can be determined solely based on the image similarity between the first neighbor i and the second neighbor i. In this case, the neighborhood matching degree of the i-th point cloud data is the image similarity between the first neighbor i and the second neighbor i. Alternatively, the neighborhood matching degree of the i-th point cloud data can be determined based on the image similarity between the first neighbor i, the second neighbor i, and other neighbor i. These other neighbor i can be the region located after projecting the i-th point cloud data onto images other than the first and second images. In this case, the neighborhood matching degree of the i-th point cloud data is the image similarity between the first neighbor i, the second neighbor i, and other neighbor i. The specific settings can be flexibly configured according to actual circumstances and are not limited.

[0301] Furthermore, the method for obtaining the neighborhood matching degree described above can be executed by one or more devices. For example, steps 1-3 can be executed by either the first or second device to determine the neighborhood matching degree of the point cloud data. Alternatively, the first and second devices can cooperate to determine the neighborhood matching degree of the point cloud data. For instance, the second device can determine the neighborhood of the point cloud data and send relevant information about that neighborhood (such as the pixel values ​​corresponding to pixels in the neighborhood, or the features corresponding to the neighborhood) to the first device, which then determines the neighborhood matching degree of the point cloud data based on this information. The specific implementation of the cooperation between the first and second devices to obtain the neighborhood matching degree can be flexibly set according to actual conditions and is not limited.

[0302] For S702:

[0303] The first device, after acquiring the neighborhood matching degree of each of the N point cloud data, can determine the noise in the N point cloud data, or in other words, determine the noisy point cloud data in the N point cloud data, based on the neighborhood matching degree of each of the N point cloud data.

[0304] For example, after determining the neighborhood matching degree of each of the N point cloud data through the above NSSD or CDF, the point cloud data with a neighborhood matching degree less than the first matching degree threshold can be identified as noise or noisy point cloud data.

[0305] For example, after determining the neighborhood matching degree of each of the N point cloud data through the above differential hashing, the point cloud data with a neighborhood matching degree greater than the second matching degree threshold can be identified as noise or noisy point cloud data.

[0306] It is understood that the N point cloud data in the embodiments of this application include at least one real point cloud data and / or at least one noisy point cloud data. The real point cloud data can be understood as the point cloud data determined by the echo generated by the target object during the radio frequency sensing process; or, the real point cloud data can be understood as point cloud data other than the noisy point cloud data. The noisy point cloud data can also be called noise points, three-dimensional noise scattering points, or other possible names, without limitation.

[0307] It's understandable, such as Figure 10 As shown, real point cloud data is projected onto multiple two-dimensional images (such as...). Figure 10 When a point cloud is projected onto an imaging plane, the regions where each projection point corresponds to the point cloud data corresponds to similar (or identical) regions of the object, resulting in a high degree of similarity. Conversely, when noisy point cloud data is projected onto multiple 2D images, the regions where each projection point corresponds to the noisy point cloud data corresponds to different regions, resulting in a low degree of similarity. It can be understood that the aforementioned similar regions are two regions on the target object that include the actual point cloud data, and these two regions have a significant overlap, or in other words, these two regions have minor differences. The aforementioned different regions can be understood as completely different regions, or as two regions with significant differences.

[0308] After the first device identifies the noise in N point cloud data, the noise can be removed or filtered; and the filtered point cloud data can be used for multimodal fusion sensing.

[0309] In summary, in this embodiment, the first device can accurately determine the noise in the N point cloud data by acquiring the neighborhood matching degree of each of the N point cloud data. Thus, the first device can use noise-filtered point cloud data in multimodal perception, thereby improving multimodal perception performance.

[0310] It is understandable that the above-described interaction between the first and second devices illustrates how the first device acquires the neighborhood matching degree of each of the N point cloud data sets. When there are multiple second devices, the first device can acquire different information from each of the multiple second devices and, based on this different information, acquire the neighborhood matching degree of each of the N point cloud data sets. The following example, using the second device #1 perceiving the target object to obtain the first image and the second device #2 perceiving the target object to obtain the second image, illustrates the interaction between the first device and the second devices #1 and #2.

[0311] In the first possible implementation, the first device can obtain the first parameter information and the first image from the second device #1, and the second parameter information and the second image from the second device #2, and determine the neighborhood matching degree of the i-th point cloud data based on the first parameter information, the second parameter information, the first image and the second image.

[0312] For example, the acquisition of parameter information, the first image, and the second image by the first device may specifically include: the second device #1 sending the first parameter information and the first image to the first device, and correspondingly, the first device receiving the first parameter information and the first image from the second device #1. The first parameter information is used to instruct the second device #1 to perform two-dimensional perception of the target object, and the first parameter information belongs to parameter information; the second device #2 sending the first parameter information and the first image to the first device, and correspondingly, receiving the second parameter information and the second image from the second device #2. The second parameter information is used to instruct the second device #2 to receive the relevant parameters for two-dimensional perception of the target object, and the second parameter information belongs to parameter information.

[0313] In the second possible implementation, the first device can obtain relevant information about the first neighborhood i from the second device #1 and relevant information about the second neighborhood i from the second device #2, and determine the neighborhood matching degree of the i-th point cloud data based on the relevant information about the first neighborhood i and the second neighborhood i.

[0314] For example, the first device requests information about the pixels in the region where N point cloud data are projected onto a two-dimensional image from the second device #1 and the second device #2. The two-dimensional image is obtained by sensing a target object. The first device receives information about the first neighborhood i from the second device #1 and information about the second neighborhood i from the second device #2.

[0315] It is understood that the aforementioned second device #1 and second device #2 can be replaced with second device and third device, or other possible names, without limitation. Furthermore, the interaction between the first device and second device #1 and second device #2 can be understood with reference to the aforementioned interaction between the first device and second device, and will not be repeated here.

[0316] It can also be understood that when the number of two-dimensional images used to determine the neighborhood matching degree increases, such as using 3 or 4 two-dimensional images to determine the neighborhood matching degree, the number of second devices can also be increased adaptively. For example, when using 3 two-dimensional images to determine the neighborhood matching degree, there can be 3 second devices, which can each provide a two-dimensional image or neighborhood information to the first device. The number of two-dimensional images and the number of second devices can be flexibly set according to the actual situation, and the specific implementation principle of having more than 2 two-dimensional images and multiple second devices can be understood by referring to the above content, and will not be repeated here.

[0317] The above, in conjunction with the method embodiments, provides an overall overview of the communication method provided in this application. A specific example is given below to illustrate the above method.

[0318] like Figure 10 As shown in Figure (a), for a building, the three-dimensional coordinates (x, y, z) of a certain wall surface are: x = -57, y ranges from [-103, 103], and z ranges from [0, 20]. The wall surface is made of brick and has multiple windows, one every 10m horizontally and one every 4m vertically. The windows are 2m x 2m in size and made of glass. Figure 10 As shown in Figure (b), two two-dimensional images of the wall surface are obtained by two second devices (optical sensing nodes). Figure 11 As shown in Figure (c), the first device performs radio frequency sensing on the wall surface to obtain multiple point cloud data. It then acquires the corresponding neighborhood matching degree for these multiple point cloud data, and based on this neighborhood matching degree, determines the noise in these multiple point cloud data, resulting in... Figure 2 Figure (d) shows the noise point cloud data in multiple point cloud datasets, where red dots represent noisy point cloud data and blue dots represent real point cloud data. Simulation results show that the method provided in the above embodiment can effectively filter noisy point cloud data from multiple point cloud datasets.

[0319] For example, Figure 11 Flowchart of the communication method provided in the embodiments of this application Figure 11 This method can be applied to the interaction between the first and second devices in the aforementioned communication system.

[0320] like Figure 7 As shown, the flow of this communication method is as follows:

[0321] S1101, the second device acquires capability parameters.

[0322] The capability parameters include information related to the capabilities of the second device. These capability parameters are used to indicate the first information that the second device is capable of reporting to the first device.

[0323] The first information is used to indicate relevant information for the second device to perceive an object, such as the internal and external parameters of the second device, or a two-dimensional image obtained by the second device from perceiving the target object. Alternatively, the first information is used to indicate information processed based on the relevant information obtained by the second device from perceiving the object, such as information obtained by processing the two-dimensional image obtained by the second device from perceiving the target object.

[0324] In one possible design, the capability parameters include at least one of the following information of the second device: information protection level information, sensing capability information, communication capability information, computing capability information, storage capability information, internal and external parameter transmission information, neighborhood acquisition information, or neighborhood matching degree acquisition information.

[0325] The neighborhood acquisition information is used to instruct the second device to support providing the first device with relevant information about pixels in a first region, which is the region where the point cloud data is projected onto the two-dimensional image.

[0326] Other information can be found in the relevant description in the aforementioned "S701", and will not be repeated here.

[0327] S1102, the second device sends capability parameters to the first device, and correspondingly, the first device receives capability parameters from the second device.

[0328] The second device can reuse existing message sending capability parameters or use newly defined message sending capability parameters. The specific settings can be flexibly configured according to the actual situation without any restrictions.

[0329] S1103, the first device sends a first message to the second device according to the capability parameters. Correspondingly, the second device receives the first message from the first device.

[0330] The first message is used to request relevant information from the second device to perceive the target object, such as the internal and external parameters of the second device, and the two-dimensional image obtained by the second device from perceiving the target object.

[0331] It's understandable that the first message will differ depending on the information included in the capability parameters. The following sections will explain these different scenarios.

[0332] Case 11.1: Capability parameters include neighborhood acquisition information.

[0333] That is, the capability parameters include neighborhood acquisition information, which is used to instruct the second device to support providing the first device with relevant information about pixels in the first region. The first region is the region where the point cloud data is projected onto the two-dimensional image. The relevant information in the first message used to request the second device to perceive the target object may specifically include: the first message is used to request relevant information about pixels in the region where N point cloud data are projected onto the two-dimensional image. The N point cloud data are three-dimensional point cloud data obtained by perceiving the target object, N is an integer greater than 1, and the two-dimensional image is obtained by perceiving the target object.

[0334] The first message may include neighborhood description information, which indicates the size and / or shape of the region where the N point cloud data are projected onto the 2D image. The first message may also include the N point cloud data. For details on the first message, please refer to the relevant introduction in "Case 7.1" above, which will not be repeated here.

[0335] Optionally, the above communication method may further include: in response to the first message, the second device sends second information to the first device, and correspondingly, the first device receives the second information from the second device, the second information being used to indicate the relevant information of pixels in the first neighborhood i and the second neighborhood i, the first neighborhood i being the region where the i-th point cloud data is projected onto the first image from the N point cloud data, the second neighborhood i being the region where the i-th point cloud data is projected onto the second image, the first image and the second image being images obtained by two-dimensional perception of the target object, the first image being different from the second image, and i being an integer from 1 to N; the first device obtains the neighborhood matching degree of the i-th point cloud data according to the second information, and determines the noise in the N point cloud data according to the neighborhood matching degree of the i-th point cloud data.

[0336] The second information may include first neighborhood information i and second neighborhood information i. The first neighborhood information i includes the pixel value corresponding to the pixel in the first neighborhood i, and the second neighborhood information i includes the pixel value corresponding to the pixel in the second neighborhood i. The second information may also include first neighborhood feature i and second neighborhood feature i. The first neighborhood feature i is used to indicate the image features of the first neighborhood i, and the second neighborhood feature i is used to indicate the image features of the second neighborhood i. The image features are related to the pixel.

[0337] It is understood that the above second information can be referred to the relevant introduction of the first information in the aforementioned "S701", and will not be repeated here.

[0338] Furthermore, when the second information includes the first neighborhood information i and the second neighborhood information i, the above communication method may further include: the first device obtains the neighborhood matching degree of the i-th point cloud data according to the first neighborhood information i, the second neighborhood information i and the matching degree calculation method, wherein the matching degree calculation method is any one of the following: normalized squared difference, cumulative density function, or neighborhood feature calculation, for details please refer to the relevant introduction in the aforementioned "Case 7.1", which will not be repeated here.

[0339] Furthermore, when the second information includes the first neighborhood information i and the second neighborhood information i, the aforementioned neighborhood acquisition information used to instruct the second device to support providing the first device with relevant information about the pixels in the first region can specifically include: neighborhood extraction acquisition information used to instruct the second device to support providing the first device with the pixel values ​​corresponding to the pixels in the first region. In this case, the neighborhood extraction acquisition information can also be referred to as neighborhood extraction acquisition information, and for details, please refer to the relevant introduction of neighborhood extraction acquisition information in the aforementioned "Case 7.1", which will not be repeated here. In addition, the capability parameters may also include neighborhood feature acquisition information, which can be referred to the relevant introduction in the aforementioned "Case 7.1", which will not be repeated here.

[0340] Furthermore, when the second information includes the first neighborhood feature i and the second neighborhood feature i, the neighborhood acquisition information used to instruct the second device to support providing the first device with relevant information about the pixels in the first region can specifically include: neighborhood extraction acquisition information used to instruct the second device to support providing the first device with image features of the first region. In this case, the neighborhood extraction acquisition information can also be referred to as neighborhood feature acquisition information, and for details, please refer to the relevant introduction of neighborhood feature acquisition information in the aforementioned "Case 7.1", which will not be repeated here. In addition, the capability parameters may also include neighborhood extraction acquisition information, which can be referred to the relevant introduction in the aforementioned "Case 7.1", which will not be repeated here.

[0341] Case 11.2: Capability parameters include neighborhood matching degree acquisition information.

[0342] That is, the capability parameters include neighborhood matching degree acquisition information, which is used to indicate that the second device supports providing the neighborhood matching degree corresponding to the point cloud data to the first device. The neighborhood matching degree corresponding to the point cloud data is used to indicate the image similarity of the region where the point cloud data is projected onto multiple two-dimensional images. The first message is used to request the relevant information of the second device to perceive the target object, which may specifically include: the first message is used to request the neighborhood matching degree corresponding to N point cloud data, where N point cloud data are three-dimensional point cloud data obtained by perceiving the target object, and N is an integer greater than 1. The neighborhood matching degree corresponding to N point cloud data is used to indicate the image similarity of the region where the i-th point cloud data in the N point cloud data is projected onto multiple two-dimensional images, where i is an integer from 1 to N. The multiple two-dimensional images are obtained by perceiving the target object.

[0343] The first message may include neighborhood description information, and the first message may also include N point cloud data. For details, please refer to the relevant introduction in "Case 11.1" above, which will not be repeated here.

[0344] Optionally, the above communication method may further include: in response to a first message, a second device sends the neighborhood matching degree of each of N point cloud data to a first device; correspondingly, the first device receives the neighborhood matching degree of each of the N point cloud data from the second device; the neighborhood matching degree of the i-th point cloud data is determined based on the image similarity between the first neighborhood i and the second neighborhood i; the first neighborhood i is the region where the i-th point cloud data is projected onto the first image, and the second neighborhood i is the region where the i-th point cloud data is projected onto the second image; the first image and the second image are images obtained by two-dimensional perception of the target object, and the first image and the second image are different; the first device determines the noise in the N point cloud data based on the neighborhood matching degree of each of the N point cloud data.

[0345] Furthermore, the above communication method may further include: a second device projecting the i-th point cloud data onto a first image and a second image; the second device determining a first neighborhood i and a second neighborhood i based on the first image and the second image mapped with the i-th point cloud data; the second device determining first neighborhood information i and second neighborhood information i based on the first neighborhood i and the second neighborhood i, wherein the first neighborhood information i includes the pixel value corresponding to the pixel in the first neighborhood i, and the second neighborhood information i includes the pixel value corresponding to the pixel in the second neighborhood i; and the second device determining the neighborhood matching degree of the i-th point cloud data based on the first neighborhood information i and the second neighborhood information i.

[0346] Furthermore, the above communication method may further include: a second device projecting the i-th point cloud data onto a first image and a second image; the second device determining a first neighborhood i and a second neighborhood i based on the first image and the second image mapped with the i-th point cloud data; the second device determining a first neighborhood feature i and a second neighborhood feature i based on the first neighborhood i and the second neighborhood i, wherein the first neighborhood feature i is used to indicate the image features of the first neighborhood i and the second neighborhood feature i is used to indicate the image features of the second neighborhood i; and the second device determining the neighborhood matching degree of the i-th point cloud data based on the first neighborhood feature i and the second neighborhood feature i.

[0347] It is understandable that the content of Case 11.2 can be referred to the relevant introduction in the aforementioned Case 7.2, and will not be repeated here.

[0348] Case 11.3: Capability parameters include internal and external parameter transmission information.

[0349] That is, the capability parameters include internal and external parameter transmission information, which is used to instruct the second device to support providing the first device with relevant parameters and images for object perception; the first message is used to request the second device to perceive the target object. The relevant information may specifically include: the first message is used to request the second device to perceive the target object (two-dimensional) with relevant parameters and at least one image.

[0350] The relevant parameters may include at least one of the following information of the second device: focal length, resolution, field of view, position, or pointing. For details, please refer to the relevant introduction in "Case 7.3" above, which will not be repeated here.

[0351] Optionally, the above communication method may further include: in response to a first message, a second device sends parameter information, a first image, and a second image to a first device; correspondingly, the first device receives parameter information, the first image, and the second image from the second device. The parameter information is used to project N point cloud data onto the first image and the second image. The N point cloud data are three-dimensional point cloud data obtained by perceiving a target object, where N is an integer greater than 1. The first image and the second image are obtained by two-dimensional perception of the target object, and the first image and the second image are different. The first device obtains the neighborhood matching degree of each of the N point cloud data according to the parameter information, the first image, and the second image, and determines the noise in the N point cloud data according to the neighborhood matching degree of each of the N point cloud data. The neighborhood matching degree of the i-th point cloud data is determined based on the image similarity between the first neighborhood i and the second neighborhood i. The first neighborhood i is the region where the i-th point cloud data is projected onto the first image, and the second neighborhood i is the region where the i-th point cloud data is projected onto the second image, where i is an integer from 1 to N.

[0352] Furthermore, the process by which the first device obtains the neighborhood matching degree of each of the N point cloud data points based on the parameter information, the first image, and the second image can specifically include: the first device projecting the i-th point cloud data point onto the first image and the second image based on the parameter information; the first device determining a first neighborhood i and a second neighborhood i based on the first image and the second image mapped with the i-th point cloud data point; the first device determining first neighborhood information i and second neighborhood information i based on the first neighborhood i and the second neighborhood i, where the first neighborhood information i includes the pixel value corresponding to the pixel point in the first neighborhood i, and the second neighborhood information i includes the pixel value corresponding to the pixel point in the second neighborhood i; and the first device determining the neighborhood matching degree of the i-th point cloud data point based on the first neighborhood information i and the second neighborhood information i.

[0353] Furthermore, the process by which the first device obtains the neighborhood matching degree of each of the N point cloud data points based on parameter information, the first image, and the second image can specifically include: the first device projecting the i-th point cloud data point onto the first image and the second image based on the parameter information; the first device determining a first neighborhood i and a second neighborhood i based on the first image and the second image mapped with the i-th point cloud data point; the first device determining a first neighborhood feature i and a second neighborhood feature i based on the first neighborhood i and the second neighborhood i, where the first neighborhood feature i indicates the image features of the first neighborhood i and the second neighborhood feature i indicates the image features of the second neighborhood i; and the first device determining the neighborhood matching degree of the i-th point cloud data point based on the first neighborhood feature i and the second neighborhood feature i.

[0354] In summary, in this embodiment, the first device can request relevant information from the second device regarding the second device's perception of the target object based on the capability parameters sent by the second device. This avoids the first device failing to request information from the second device, thus preventing additional communication overhead.

[0355] Understandable. Figure 7 The illustrated embodiments and Figure 11 The embodiments shown are similar, and the same parts can be understood by referring to each other, and will not be described again here. In addition, in the various embodiments of this application, unless otherwise specified or logically conflicting, the terminology and / or descriptions between different embodiments are consistent and can be referenced by each other. The technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships.

[0356] It is also understood that in the various embodiments of this application, "two-dimensional" can also be referred to as "2D" and "three-dimensional" can also be referred to as "3D". Furthermore, in the various embodiments of this application, the names of each message, each piece of information, and each parameter are merely illustrative representations, and each message, each piece of information, and each parameter can be replaced with any possible representation without limitation.

[0357] The above combinationFigures 12-15 and Figure 12 The illustrated method embodiments provide an overall overview of the communication method flow provided in this application. For ease of understanding, the following will combine... Figure 3 The interaction process between the first device and the second device is described in detail through method embodiments.

[0358] Scene 1:

[0359] Figure 12 Flowchart of the communication method provided in the embodiments of this application Figure 7 This communication method can be applied to the interaction between the first and second devices in the aforementioned communication system. In scenario 1, the second device supports the transmission of internal and external parameters and two-dimensional images, meaning that the second device (or the device containing the second device) has the capability to transmit internal and external parameters. In this case, the first device obtains the internal and external parameters of the second device and at least one two-dimensional image obtained from the perception of the target object from the second device, and based on the internal and external parameters and the at least one two-dimensional image, obtains the neighborhood matching degree of each of N point cloud data, where N is an integer greater than 1.

[0360] like Figure 11 As shown, the flow of this communication method is as follows:

[0361] S1201, the first device sends a multimodal sensing request message to the second device. Correspondingly, the second device receives the multimodal sensing request message from the first device.

[0362] The multimodal sensing request message is used to request the second device to participate in multimodal sensing. Alternatively, the multimodal sensing request message is used to request the second device to report its capability parameters.

[0363] S1202, the second device sends capability parameters to the first device. Correspondingly, the first device receives the capability parameters from the second device.

[0364] In response to the multimodal perception request message, the second device sends capability parameters to the first device.

[0365] This capability parameter corresponds to Figure 7 The fourth capability parameter in the illustrated embodiment. Alternatively, the capability parameter corresponds to... Figure 11 The capability parameters in “Case 11.3” in the illustrated embodiment.

[0366] S1203, the first device sends request message #1 to the second device. Correspondingly, the second device receives request message #1 from the second device.

[0367] Request message #1 corresponds to Figure 7 The third message in the illustrated embodiment. Or, request message #1 corresponds to Figure 7The first message in “Case 11.3” in the illustrated embodiment.

[0368] S1204, the second device sends internal and external parameters and a two-dimensional image to the first device. Correspondingly, the first device receives the internal and external parameters and the two-dimensional image from the second device.

[0369] In response to request message #1, the second device sends internal and external parameters and a two-dimensional image to the first device.

[0370] Internal and external parameters correspond Figure 11 The parameter information in the illustrated embodiment; the two-dimensional image includes two two-dimensional images, which correspond to... Figure 11 The first and second images in the illustrated embodiment.

[0371] Or, internal and external parameters correspond Figure 7 The parameter information in "Case 11.3" of the illustrated embodiment; the two-dimensional image includes two two-dimensional images, which correspond to... Figure 11 The first and second images in “Case 11.3” of the illustrated embodiment.

[0372] S1205, the first device calculates the neighborhood matching degree corresponding to N point cloud data based on internal and external parameters and two-dimensional images.

[0373] S1206, the first device determines the noise in the N point cloud data based on the neighborhood matching degree of each of the N point cloud data.

[0374] It is understandable that S1201-S1206 can refer to the aforementioned. Figure 13 or Figure 4 The relevant descriptions in the illustrated embodiments will not be repeated here. Furthermore, S1201 and S1203 are optional steps; for example, the second device may periodically report capability information to the first device, and / or the second device may periodically report internal and external parameters and two-dimensional images to the first device.

[0375] Scene 2:

[0376] Figure 13 Flowchart of the communication method provided in the embodiments of this application Figure 7 This communication method can be applied to the interaction between the first and second devices in the aforementioned communication system. In scenario 2, the second device supports neighborhood extraction and transmission of extracted neighborhood information, meaning the second device (or the device containing the second device) has the capability to extract and transmit extracted neighborhood information. In this case, the first device obtains the neighborhood information of the second device from the second device and, based on this neighborhood information, obtains the neighborhood matching degree of each of the N point cloud data, where N is an integer greater than 1.

[0377] likeFigure 11 As shown, the flow of this communication method is as follows:

[0378] S1301, the first device sends a multimodal sensing request message to the second device. Correspondingly, the second device receives the multimodal sensing request message from the first device.

[0379] The specific implementation principle of S1301 can be found in the aforementioned introduction of S1201, and will not be repeated here.

[0380] S1302, the second device sends capability parameters to the first device. Correspondingly, the first device receives the capability parameters from the second device.

[0381] In response to the multimodal perception request message, the second device sends capability parameters to the first device.

[0382] This capability parameter corresponds to Figure 7 The first capability parameter in the illustrated embodiment. Alternatively, the capability parameter corresponds to... Figure 11 In the illustrated embodiment, the capability parameter in “Case 11.1” includes neighborhood acquisition information, which is used to instruct the second device to support providing the first device with the pixel values ​​corresponding to the pixels in the first region.

[0383] S1303, the first device sends request message #1 to the second device. Correspondingly, the second device receives request message #1 from the second device.

[0384] Request message #1 includes at least one of the following: N point cloud data or neighborhood description information.

[0385] Request message #1 corresponds to Figure 7 The first message in the illustrated embodiment. Alternatively, request message #1 corresponds to... Figure 11 The first message in “Case 11.1” in the illustrated embodiment.

[0386] S1304, the second device obtains multiple neighborhood information according to request message #1.

[0387] For example, according to request message #1, the second device projects N point cloud data onto multiple two-dimensional images obtained for the target object perception to obtain multiple projection points; and extracts the neighborhood information of these multiple projection points, which is the pixel value corresponding to the pixel point in the region.

[0388] S1305, the second device sends multiple neighborhood information to the first device. Correspondingly, the first device receives the neighborhood information from the second device.

[0389] Neighborhood information includes the pixel values ​​of the pixels in the region where the projection point of the point cloud data is located. It can be understood that the second device can send multiple neighborhood information messages in the order in which it receives N point cloud data messages; alternatively, the second device can use an index to send multiple neighborhood information messages. It can be understood that neighborhood information corresponds to... Figure 7 The illustrated embodiments or Figure 11 Neighborhood information in “Case 11.1” in the illustrated embodiment.

[0390] Optionally, the second device sends multiple neighborhood information to the first device, including: the second device sending N point cloud data and multiple neighborhood information to the first device. The N point cloud data and the multiple neighborhood information have a corresponding relationship; for example, the point cloud data with the corresponding relationship and the multiple neighborhood information can be placed in a tuple. In this way, the first device can determine the correspondence between the N point cloud data and the multiple neighborhood information based on the received N point cloud data and multiple neighborhood information.

[0391] S1306, the first device calculates the neighborhood matching degree corresponding to N point cloud data based on multiple neighborhood information.

[0392] S1307, based on the neighborhood matching degree of each of the N point cloud data, determine the noise in the N point cloud data.

[0393] It is understandable that S1301-S1307 can refer to the aforementioned. Figure 14 or Figure 5 The relevant descriptions in the illustrated embodiments will not be repeated here. Furthermore, S1301 is an optional step; for example, the second device may periodically report capability information to the first device.

[0394] Scene 3:

[0395] Figure 14 Flowchart of the communication method provided in the embodiments of this application Figure 7 This communication method can be applied to the interaction between the first and second devices in the aforementioned communication system. In scenario 3, the second device supports neighborhood feature extraction and transmission of the extracted neighborhood features, meaning the second device (or the device containing the second device) has the capability to extract and transmit the extracted neighborhood features. In this case, the first device obtains the neighborhood features of the second device from the second device and, based on these neighborhood features, obtains the neighborhood matching degree of each of the N point cloud data points, where N is an integer greater than 1.

[0396] like Figure 11 As shown, the flow of this communication method is as follows:

[0397] S1401, the first device sends a multimodal sensing request message to the second device. Correspondingly, the second device receives the multimodal sensing request message from the first device.

[0398] The specific implementation principle of S1401 can be found in the aforementioned introduction of S1201, and will not be repeated here.

[0399] S1402, the second device sends capability parameters to the first device. Correspondingly, the first device receives the capability parameters from the second device.

[0400] In response to the multimodal perception request message, the second device sends capability parameters to the first device.

[0401] This capability parameter corresponds to Figure 7 The second capability parameter in the illustrated embodiment. Alternatively, the capability parameter corresponds to... Figure 11 In the illustrated embodiment, the capability parameter in “Case 11.1” includes neighborhood acquisition information, which is used to instruct the second device to support providing image features of the first region to the first device.

[0402] S1403, the first device sends request message #1 to the second device. Correspondingly, the second device receives request message #1 from the second device.

[0403] Request message #1 includes at least one of the following: N point cloud data or neighborhood description information.

[0404] Request message #1 corresponds to Figure 7 The first message in the illustrated embodiment. Alternatively, request message #1 corresponds to... Figure 11 The first message in “Case 11.1” in the illustrated embodiment.

[0405] S1404, the second device obtains multiple neighborhood information according to request message #1.

[0406] For example, according to request message #1, the second device projects N point cloud data onto multiple two-dimensional images obtained for the target object perception to obtain multiple projection points; and extracts the neighborhood information of these multiple projection points, which is the pixel value corresponding to the pixel point in the region.

[0407] S1405, the second device calculates multiple neighborhood features based on multiple neighborhood information.

[0408] Neighborhood information can be found in the relevant description in S1305 above, and will not be repeated here.

[0409] Multiple neighborhood features are features extracted based on information from multiple neighborhoods. This can be understood as neighborhood features corresponding to... Figure 7 The illustrated embodiments or Figure 11 The neighborhood features in “Case 11.1” in the illustrated embodiment.

[0410] S1406, the second device sends multiple neighborhood features to the first device. Correspondingly, the first device receives multiple neighborhood features from the second device.

[0411] The second device can send multiple neighborhood features in the order of receiving N point cloud data; alternatively, the second device can send multiple neighborhood features using an index. It can be understood that neighborhood information corresponds to... Figure 15 The illustrated embodiments or Figure 5 Neighborhood information in “Case 11.1” in the illustrated embodiment.

[0412] Optionally, the second device sends multiple neighborhood features to the first device, including: the second device sending N point cloud data and multiple neighborhood features to the first device. The N point cloud data and the multiple neighborhood features have a corresponding relationship; for example, the point cloud data with the corresponding relationship and the multiple neighborhood features can be placed in a tuple. In this way, the first device can determine the correspondence between the N point cloud data and the multiple neighborhood features based on the received N point cloud data and multiple neighborhood features.

[0413] S1407, the first device calculates the neighborhood matching degree corresponding to N point cloud data based on multiple neighborhood features.

[0414] S1408, the first device determines the noise in the N point cloud data based on the neighborhood matching degree of each of the N point cloud data.

[0415] It is understandable that S1401-S1408 can refer to the aforementioned. Figure 15 or Figure 7 The relevant descriptions in the illustrated embodiments will not be repeated here. Furthermore, S1401 is an optional step; for example, the second device may periodically report capability information to the first device.

[0416] Scene 4:

[0417] Figure 11 Flowchart of the communication method provided in the embodiments of this application Figure 7 This communication method can be applied to the interaction between the first and second devices in the aforementioned communication system. In scenario 4, the second device supports providing neighborhood matching degrees to other devices, that is, the second device (or the device in which the second device is located) has the ability to provide neighborhood matching degrees to other devices. In this case, the first device obtains the neighborhood matching degrees of each of the N point cloud data from the second device, where N is an integer greater than 1.

[0418] like Figure 11 As shown, the flow of this communication method is as follows:

[0419] S1501, the first device sends a multimodal sensing request message to the second device. Correspondingly, the second device receives the multimodal sensing request message from the first device.

[0420] The specific implementation principle of S1501 can be found in the aforementioned introduction of S1201, and will not be repeated here.

[0421] S1502, the second device sends capability parameters to the first device. Correspondingly, the first device receives the capability parameters from the second device.

[0422] In response to the multimodal perception request message, the second device sends capability parameters to the first device.

[0423] This capability parameter corresponds to Figure 7 The third capability parameter in the illustrated embodiment. Alternatively, the capability parameter corresponds to... Figure 11 The capability parameters in “Case 11.2” of the illustrated embodiment.

[0424] S1503, the first device sends request message #1 to the second device. Correspondingly, the second device receives request message #1 from the second device.

[0425] Request message #1 includes at least one of the following: N point cloud data or neighborhood description information.

[0426] Request message #1 corresponds to Figures 12-15 The second message in the illustrated embodiment. Alternatively, request message #1 corresponds to... Figures 12-15 The first message in “Case 11.2” in the illustrated embodiment.

[0427] Optionally, request message #1 may also include a two-dimensional image, which is an image obtained by a device other than the second device from sensing the target object.

[0428] S1504, the second device calculates the neighborhood matching degree corresponding to N point cloud data according to request message #1.

[0429] S1505, the second device sends the neighborhood matching degree of each of the N point cloud data to the first device. Correspondingly, the first device receives the neighborhood matching degree of each of the N point cloud data from the second device.

[0430] The second device can send multiple neighborhood matching degrees in the order of receiving N point cloud data; or, the second device can send multiple neighborhood matching degrees using an index.

[0431] Optionally, the second device sends multiple neighborhood matching degrees to the first device, including: the second device sending N point cloud data and multiple neighborhood matching degrees to the first device. The N point cloud data and the multiple neighborhood matching degrees have a corresponding relationship; for example, the point cloud data with the corresponding relationship and the multiple neighborhood matching degrees can be placed in a tuple. In this way, the first device can determine the correspondence between the N point cloud data and the multiple neighborhood matching degrees based on the received N point cloud data and multiple neighborhood matching degrees.

[0432] S1506, the first device determines the noise in the N point cloud data based on the neighborhood matching degree of each of the N point cloud data.

[0433] It is understandable that S1501-S1506 can refer to the aforementioned. Figures 7-15 or Figures 16-17 The relevant descriptions in the illustrated embodiments will not be repeated here. Furthermore, S1501 is an optional step; for example, the second device may periodically report capability information to the first device.

[0434] It can also be understood that in the above scenarios 1-4, the second device can report its capabilities to the first device through multiple messages. For example, the second device can first report information protection level information, sensing capability information, communication capability information, computing capability information, or storage capability information to the first device, and then report other capabilities to the first device.

[0435] It's also understandable that scenarios 1-3 above are all based on a single second device. When there are multiple second devices, the above process can be adapted. For example, when there are two second devices, the first device sends multimodal perception request messages to both second devices respectively, which will not be elaborated here. Furthermore, when there are multiple second devices, after receiving the capabilities reported by multiple second devices, the first device can select the second device with the strongest overall capabilities to perform subsequent operations.

[0436] In addition, the above Figure 16 The embodiment shown is merely an example; in different situations, Figure 1 The steps in the illustrated embodiments can be varied accordingly without limitation.

[0437] The above combination Figure 16 The communication method provided in the embodiments of this application is described in detail below. Figure 16 This document describes in detail the communication apparatus used to perform the communication method provided in the embodiments of this application.

[0438] Figures 7-15 This is a schematic diagram of the structure of the communication device provided in the embodiments of this application. Figures 7-15 For example, such as Figure 16As shown, the communication device 1600 includes a transceiver module 1601 and a processing module 1602. For ease of explanation, Figure 16 Only the main components of the communication device are shown.

[0439] The transceiver module 1601 is used to perform the above. Figure 16 The sending and receiving functions of the method shown are executed by the processing module 1602. Figures 7-15 The method shown includes functions other than sending and receiving.

[0440] Optionally, the transceiver module 1601 may include a transmitting module ( Figures 7-15 (not shown in the image) and receiving module ( Figure 17 (Not shown in the diagram). The transmitting module implements the transmitting function of the communication device 1600, and the receiving module implements the receiving function of the communication device 1600.

[0441] Optionally, the communication device 1600 may also include a storage module. Figure 2 (Not shown in the image), the storage module stores programs or instructions. When the processing module 1602 executes the program or instructions, the communication device 1600 can perform the above-described method. Figure 17 The method shown describes the functions of the terminal device or network device (such as the first device, the second device, etc.).

[0442] It is understood that the communication device 1600 may be a terminal device or a network device, or it may be a chip (system) or other component or assembly that can be disposed in the terminal device or the network device, or it may be a device that includes the terminal device or the network device. This application does not limit it in this respect.

[0443] In addition, the technical effects of the communication device 1600 can be referenced. Figure 17 The technical effects of the communication method shown will not be elaborated here.

[0444] Figure 17 Schematic diagram of the communication device provided in the embodiments of this application Figure 17 For example, the communication device may be a terminal device or a network device, or it may be a chip (system) or other component or assembly that can be disposed in the terminal device or the network device. Figure 17 As shown, the communication device 1700 may include a processor 1701. Optionally, the communication device 1700 may also include a memory 1702 and / or a transceiver 1703. The processor 1701 is coupled to the memory 1702 and the transceiver 1703, for example, they may be connected via a communication bus.

[0445] The following is combined with Figure 17 A detailed description of each component of the communication device 1700 is provided below:

[0446] The processor 1701 is the control center of the communication device 1700. It can be a single processor or a collective term for multiple processing elements. For example, the processor 1701 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement the embodiments of this application, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0447] Optionally, the processor 1701 can perform various functions of the communication device 1700, such as performing the communication method described above, by running or executing software programs stored in the memory 1702 and calling data stored in the memory 1702.

[0448] In a specific implementation, as one example, the processor 1701 may include one or more CPUs, for example... Figure 17 CPU0 and CPU1 are shown in the diagram.

[0449] In a specific implementation, as one example, the communication device 1700 may also include multiple processors, for example... Figure 17 The processors 1701 and 1704 are shown. Each of these processors can be a single-core processor or a multi-core processor. Here, "processor" can refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0450] The memory 1702 is used to store the software program that executes the solution of this application, and is controlled by the processor 1701 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0451] Optionally, the memory 1702 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 1702 may be integrated with the processor 1701 or may exist independently, and may be connected via the interface circuit of the communication device 1700. ​ (Not shown in the image) is coupled to processor 1701, and this embodiment of the application does not specifically limit this.

[0452] Transceiver 1703 is used for communication with other communication devices. For example, if communication device 1700 is a terminal, transceiver 1703 can be used to communicate with a network device or with another terminal device. As another example, if communication device 1700 is a network device, transceiver 1703 can be used to communicate with a terminal or with another network device.

[0453] Alternatively, transceiver 1703 may include a receiver and a transmitter. ​ (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the sending function.

[0454] Alternatively, the transceiver 1703 can be integrated with the processor 1701, or it can exist independently and be connected via the interface circuit of the communication device 1700. ​ (Not shown in the image) is coupled to processor 1701, and this embodiment of the application does not specifically limit this.

[0455] Understandable ​ The structure of the communication device 1700 shown does not constitute a limitation on the communication device. Actual communication devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0456] Furthermore, the technical effects of the communication device 1700 can be referred to the technical effects of the method described in the above method embodiments, and will not be repeated here.

[0457] It should be understood that the processor in the embodiments of this application can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0458] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0459] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0460] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0461] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0462] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0463] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0464] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0465] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0466] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0467] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0468] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0469] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A communication method characterized by comprising: The method applied to a first device comprises: obtaining a neighborhood matching degree of each of N point cloud data, N being an integer greater than 1, i being an integer traversing 1 to N, the N point cloud data being three-dimensional point cloud data obtained by sensing a target object, the neighborhood matching degree of an i-th point cloud data in the N point cloud data being determined according to an image similarity between a first neighborhood i and a second neighborhood i, the first neighborhood i being a region where the i-th point cloud data is located after being projected onto a first image, the second neighborhood i being a region where the i-th point cloud data is located after being projected onto a second image, the first image and the second image being two-dimensional images obtained by sensing the target object, the first image being different from the second image; determining noise in the N point cloud data according to the neighborhood matching degree of each of the N point cloud data.

2. The method of claim 1, wherein, The obtaining of the neighborhood matching degree of each of the N point cloud data comprises: obtaining first information, the first information being used to indicate related information of pixel points in the first neighborhood i and the second neighborhood i; determining the neighborhood matching degree of the i-th point cloud data according to the first information.

3. The method of claim 2, wherein, The obtaining of the first information comprises: sending a first message to a second device, the first message being used to request related information of pixel points in regions where the N point cloud data is located after being projected onto two-dimensional images, the two-dimensional images being obtained by sensing the target object; receiving the first information from the second device.

4. The method of claim 3, wherein, The first message comprises neighborhood description information, the neighborhood description information being used to indicate sizes and / or shapes of the regions where the N point cloud data is located after being projected onto the two-dimensional images.

5. The method according to any one of claims 2-4, characterized in that, The first information comprises first neighborhood information i and second neighborhood information i, the first neighborhood information i comprising pixel values corresponding to pixel points in the first neighborhood i, the second neighborhood information i comprising pixel values corresponding to pixel points in the second neighborhood i; alternatively, the first information comprises first neighborhood features i and second neighborhood features i, the first neighborhood features i being used to indicate image features of the first neighborhood i, the second neighborhood features i being used to indicate image features of the second neighborhood i, the image features being related to pixel points.

6. The method of claim 5, wherein, In a case where the first information comprises the first neighborhood information i and the second neighborhood information i, the method further comprises: receiving a first capability parameter from a second device, the first capability parameter being used to indicate that the second device supports providing pixel values corresponding to pixel points in a first region to the first device, the first region being a region where point cloud data is located after being projected onto a two-dimensional image; the sending of the first message to the second device comprises: sending the first message to the second device according to the first capability parameter.

7. The method of claim 5, wherein, In a case where the first information comprises the first neighborhood features i and the second neighborhood features i, the method further comprises: receiving a second capability parameter from a second device, the second capability parameter being used to indicate that the second device supports providing neighborhood features to the first device, the neighborhood features being used to indicate image features of a region where point cloud data is located after being projected onto a two-dimensional image; The sending the first message to the second device comprises: According to the second capability parameter, the first message is sent to the second device.

8. The method of claim 1, wherein, The obtaining the respective neighborhood matching degrees of the N point cloud data comprises: The sending the second message to the second device comprises: The first device receives the respective neighborhood matching degrees of the N point cloud data from the second device.

9. The method of claim 8, wherein, Before the sending the second message to the second device, the method further comprises: The receiving the third capability parameter from the second device comprises: The sending the second message to the second device comprises: According to the third capability parameter, the second message is sent to the second device.

10. The method of claim 1, wherein, The obtaining the respective neighborhood matching degrees of the N point cloud data comprises: The obtaining the parameter information, the first image and the second image comprises: According to the parameter information, the first image and the second image, the respective neighborhood matching degrees of the N point cloud data are determined.

11. The method of claim 10, wherein, The obtaining the parameter information, the first image and the second image comprises: The receiving the parameter information, the first image and the second image from the second device comprises:

12. The method of claim 11, wherein, Before the receiving the parameter information, the first image and the second image from the second device, the method further comprises: The receiving the fourth capability parameter from the second device comprises: According to the fourth capability parameter, the third message is sent to the second device. The method applied to the second device comprises:

13. A method of communication, comprising: The receiving the first message from the first device comprises: ​ In response to the first message, first information is sent to the first device, the first information being used to indicate related information of pixels in a first neighborhood i and a second neighborhood i, the first neighborhood i being a region where the i-th point cloud data in the N point cloud data is located after being projected to a first image, the second neighborhood i being a region where the i-th point cloud data is located after being projected to a second image, the first image and the second image being obtained by two-dimensional perception of the target object, the first image and the second image being different, and i being an integer traversing 1 to N.

14. The method of claim 13, wherein, The first message includes neighborhood description information, and the neighborhood description information is used to indicate the size and / or shape of the region where the N point cloud data is located after being projected to a two-dimensional image.

15. The method according to claim 13 or 14, characterized in that, The first information includes first neighborhood information i and second neighborhood information i, the first neighborhood information i including pixel values corresponding to pixel points in the first neighborhood i, and the second neighborhood information i including pixel values corresponding to pixel points in the second neighborhood i. Alternatively, The first information includes first neighborhood feature i and second neighborhood feature i, the first neighborhood feature i being used to indicate image features of the first neighborhood i, and the second neighborhood feature i being used to indicate image features of the second neighborhood i, the image features being related to pixel points.

16. The method of claim 15, wherein, In the case where the first information includes the first neighborhood information i and the second neighborhood information i, the method further includes: The first capability parameter is sent to the first device, and the first capability parameter is used to indicate that the second device supports providing pixel values corresponding to pixel points in a first region to the first device, the first region being a region where point cloud data is located after being projected to a two-dimensional image.

17. The method of claim 15, wherein, In the case where the first information includes the first neighborhood feature i and the second neighborhood feature i, the method further includes: The second capability parameter is sent to the first device, and the second capability parameter is used to indicate that the second device supports providing neighborhood features to the first device, the neighborhood features being used to indicate image features of a region where point cloud data is located after being projected to a two-dimensional image.

18. A method of communication, comprising: The method applied to the second device includes: A second message is received from the first device, the second message being used to request neighborhood matching degrees corresponding to N point cloud data, the N point cloud data being three-dimensional point cloud data obtained by perception of a target object, N being an integer greater than 1, and the neighborhood matching degrees corresponding to the N point cloud data being used to indicate image similarity degrees of regions where the i-th point cloud data is located after being projected to a plurality of two-dimensional images, the plurality of two-dimensional images being obtained by perception of the target object, and i being an integer traversing 1 to N. In response to the second message, the N point cloud data respective neighborhood matching degrees are sent to the first device, the neighborhood matching degree of the ith point cloud data is determined according to the image similarity between the first neighborhood i and the second neighborhood i, the first neighborhood i is the region where the ith point cloud data is located after being projected to the first image, the second neighborhood i is the region where the ith point cloud data is located after being projected to the second image, the first image and the second image are obtained by two-dimensional perception of the target object, and the first image and the second image are different.

19. The method of claim 18, wherein, Before the receiving of the second message from the first device, the method further includes: A third capability parameter is sent to the first device, the third capability parameter is used to indicate that the second device supports providing the neighborhood matching degree corresponding to the point cloud data to the first device, and the neighborhood matching degree corresponding to the point cloud data is used to indicate the image similarity of the region where the point cloud data is located after being projected to a plurality of two-dimensional images.

20. A method of communication, comprising: Applied to the second device, the method includes: Parameter information, a first image and a second image are obtained, the parameter information is used to indicate the related parameters of two-dimensional perception of the target object by the second device, the first image and the second image are obtained by two-dimensional perception of the target object by the second device, and the first image and the second image are different. The parameter information, the first image and the second image are sent to the first device.

21. The method of claim 20, wherein, The method includes: A third message is received from the first device, the third message is used to request the related parameters of two-dimensional perception of the target object by the second device and at least one image; The sending of the parameter information, the first image and the second image to the first device includes: In response to the third message, the parameter information, the first image and the second image are sent to the first device.

22. The method of claim 20 or 21, wherein, Before the sending of the parameter information, the first image and the second image to the first device, the method further includes: A fourth capability parameter is sent to the first device, the fourth capability parameter is used to indicate that the second device supports providing the related parameters of two-dimensional perception of the object and the images to the first device.

23. A communications device, characterized by The communication device is used to execute the communication method in any one of claims 1-22.

24. A communications device, characterized by Including: A processor and a memory; The memory is used to store computer instructions, when the processor executes the instructions, so that the communication device executes the communication method in any one of claims 1-22.

25. A computer readable storage medium, characterized in that, The computer readable storage medium includes computer programs or instructions, when the computer programs or instructions run on the communication device, so that the communication device executes the method in any one of claims 1-22.

26. A computer program product, characterised in that, The computer program product includes computer programs or instructions, when the computer programs or instructions are run by the communication device, so that the method in any one of claims 1-22 is executed.

Images