Sensor fusion methods, devices, storage media and software products

By preprocessing and fusing the raw data from non-3GPP sensing devices, the problem of poor interoperability of sensing devices in the sensing network is solved, realizing a wide-area coverage sensing system and improving networking and sensing capabilities.

CN123123976APending Publication Date: 2026-11-17ZTE CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202510638944.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-05-16
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing sensing networks lack effective networking solutions when accessing non-3GPP sensing devices, resulting in poor interoperability of sensing devices, limited sensing distance, and high cost for wide-area coverage, making it impossible to form a wide-area coverage sensing system.

Method used

By receiving control information, the first and second sensing data are fused, and the raw data collected by the sensing devices is preprocessed to meet the requirements of the sensor network. This reduces the processing complexity of the sensor network on the data collected by the sensing devices and concentrates computing power on network fusion.

Benefits of technology

It enables flexible networking of sensor networks and large-scale sensing devices, improves networking and sensing capabilities, reduces computational complexity, and enhances wide-area coverage sensing systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123123976A_ABST
    Figure CN123123976A_ABST
Patent Text Reader

Abstract

This disclosure provides a sensing fusion method, apparatus, storage medium, and program product, relating to the field of sensory integration technology, for improving the networking and sensing capabilities of sensory networks. The method includes: receiving control information, the control information being used to instruct the fusion of first sensing data and second sensing data; wherein the first sensing data originates from at least one sensor node, and the second sensing data is data that meets the requirements of a sensory network, obtained by preprocessing raw data collected by at least one sensing device; and fusing the first sensing data and the second sensing data based on the control information to obtain fused sensing data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to the field of sensor integration technology, and in particular to a sensing fusion method, device, storage medium and program product. Background Technology

[0002] Existing non-3rd Generation Partnership Project (3GPP) sensing devices / networks, such as multiple cameras at intersections or multiple sensors on vehicles, are still in the local network stage due to sensing limitations, short sensing distance of individual sensing devices (cameras can only sense up to 300m), poor interoperability between different sensing devices, and high costs for wide-area coverage. It is necessary to build a wide-area coverage sensing system based on a sensor network, leveraging the existing communication and computing networks of the sensor network, and networking non-3GPP sensing devices with the sensor network.

[0003] Current sensor network topology designs primarily achieve networking between multiple sensor stations through information exchange. When non-3GPP sensing devices are added, their access to the sensor network mainly occurs through direct access by a single 3GPP sensing device, including direct access to the base station sensing unit or access to the sensing unit via the public network. The networking issues between the sensor network and non-3GPP sensing devices are not considered. Summary of the Invention

[0004] This disclosure provides a sensing fusion method, apparatus, storage medium, and program product to enhance the networking and sensing capabilities of sensor networks.

[0005] To achieve the above objectives, the present disclosure adopts the following technical solution.

[0006] On the one hand, a perceptual fusion method is provided and applied to the first node. This method includes:

[0007] Receive control information, which is used to instruct the fusion of first sensing data and second sensing data; wherein, the first sensing data comes from at least one sensing node, and the second sensing data is data that meets the requirements of the sensing network, which is obtained by preprocessing the raw data collected by at least one sensing device.

[0008] Based on control information, the first and second sensing data are fused to obtain fused sensing data.

[0009] On the other hand, a perceptual fusion method is provided and applied to a first computing node, the method comprising:

[0010] Receive control information, which is used to instruct the fusion of first sensing data and second sensing data; wherein, the first sensing data comes from at least one sensing node, and the second sensing data is data that meets the requirements of the sensing network, which is obtained by preprocessing the raw data collected by at least one sensing device.

[0011] Perception fusion operations are performed based on control information.

[0012] On another front, a communication device is provided for use in a first node, comprising:

[0013] The communication module receives control information, which is used to instruct the fusion of first sensing data and second sensing data; wherein, the first sensing data comes from at least one sensing node, and the second sensing data is data that meets the requirements of the sensing network, obtained by preprocessing the raw data collected by at least one sensing device.

[0014] The processing module fuses the first and second sensing data based on control information to obtain fused sensing data.

[0015] In another aspect, a communication device is provided for use in a first computing node, comprising:

[0016] The communication module is used to receive control information, which is used to instruct the fusion of first sensing data and second sensing data; wherein, the first sensing data comes from at least one sensing node, and the second sensing data is data that meets the requirements of the sensing network, obtained by preprocessing the raw data collected by at least one sensing device.

[0017] The processing module is used to perform perception fusion operations based on control information.

[0018] In another aspect, a communication device is provided, comprising: a memory and a processor; the memory and the processor are coupled; the memory is used to store computer program instructions executable by the processor; and the processor executes the computer program instructions to implement the method of any of the above embodiments.

[0019] In another aspect, a computer-readable storage medium is provided, on which computer program instructions are stored, which, when executed on a computer (e.g., a communication device), implement the methods of any of the above embodiments.

[0020] In another aspect, a computer program product is provided, the computer program product including computer program instructions that, when executed, implement the method of any of the above embodiments.

[0021] The embodiments provided in this disclosure fuse first sensing data and second sensing data based on received control information to obtain fused sensing data. The first sensing data originates from at least one sensing node, and the second sensing data is data that meets the requirements of a sensing network, obtained by preprocessing raw data collected by at least one sensing device. In this way, the sensing network does not need to have the processing capability for raw data collected by various sensing devices during the sensing data fusion process, reducing the computational complexity of processing data collected by sensing devices. This allows the sensing network to concentrate its computing power on network fusion rather than performing basic processing on the raw data collected by each sensing device, thereby achieving the fusion of data collected by a large number of sensing devices and sensing data from sensing nodes, and improving the networking and sensing capabilities of the sensing network. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the architecture of a communication system provided in an embodiment of the present disclosure;

[0023] Figure 2 A flowchart of a perception fusion method provided in an embodiment of this disclosure;

[0024] Figure 3 A schematic diagram of a sensor network configuration provided in an embodiment of this disclosure;

[0025] Figure 4 An interactive flowchart of a perception fusion method provided in an embodiment of this disclosure;

[0026] Figure 5 An interactive flowchart of another perception fusion method provided in an embodiment of this disclosure;

[0027] Figure 6 A schematic diagram of a sensing scene provided in an embodiment of this disclosure;

[0028] Figure 7 A schematic diagram of another sensing scenario provided by an embodiment of this disclosure;

[0029] Figure 8 A schematic diagram of yet another sensing scenario provided by an embodiment of this disclosure;

[0030] Figure 9 A flowchart of another perception fusion method provided in this disclosure embodiment;

[0031] Figure 10 This is a schematic diagram of the structure of a communication device provided in an embodiment of the present disclosure;

[0032] Figure 11 This is a schematic diagram of another communication device provided in an embodiment of the present disclosure;

[0033] Figure 12 This is a schematic diagram of the structure of another communication device provided in an embodiment of the present disclosure. Detailed Implementation

[0034] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this disclosure.

[0035] Unless the context otherwise requires, throughout the specification and claims, the term "comprise" and other forms such as the third-person singular "comprises" and the present participle "comprising" are interpreted as open-ended and encompassing, meaning "including, but not limited to." In the description of the specification, terms such as "one embodiment," "some embodiments," "exemplary embodiments," "example," "specific example," or "some examples" are intended to indicate that a particular feature, structure, material, or characteristic associated with that embodiment or example is included in at least one embodiment or example of this disclosure. The illustrative representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, a particular feature, structure, material, or characteristic may be included in any suitable manner in any one or more embodiments or examples.

[0036] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this disclosure, unless otherwise stated, "a plurality of" means two or more.

[0037] In this disclosure, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in this disclosure should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0038] In addition, the use of "based on" implies openness and inclusivity, because processes, steps, calculations or other actions "based on" one or more conditions or values ​​can in practice be based on additional conditions or values ​​beyond those conditions.

[0039] Communication-sensing integration (or simply sensing integration / sensing) has become an important evolutionary direction for future mobile communication technologies, with broad application prospects. Communication-sensing integration achieves both wireless sensing and wireless communication functions by sharing spectrum and hardware / software resources. A typical communication-sensing integration network architecture (or simply sensing network) consists of a wireless access network, user equipment, and a core network with sensing servers. Leveraging the sensing capabilities and widespread deployment of base stations in the sensing network, it will have wide-area coverage sensing capabilities. Besides utilizing 3GPP (3rd Generation Partnership Project) radio spectrum resources for sensing, a large number of non-3GPP sensing devices (or simply sensing devices) have been applied in the industry, such as roadside stations for vehicle-to-everything (V2X) communication, video surveillance cameras, and IoT sensing units. These multimodal sensing devices utilize local computing devices for sensing processing, forming a large number of edge computing nodes. Integrating these multimodal sensing devices and edge computing nodes into a network architecture that integrates communication and sensing can not only improve the sensing capabilities (sensing accuracy, sensing range, etc.) and computing capabilities of the sensing network, but also enable the fusion of various non-3GPP sensing devices through the sensing network, thus realizing a complete cross-multimodal wide-area sensing network.

[0040] Existing non-3GPP sensing devices / networks, such as the fusion of multiple cameras at intersections or the fusion of multiple sensors on vehicles, are still in the local network stage due to the sensing limitations of the sensing devices, the short sensing distance of a single sensing device (the maximum sensing distance of a camera is 300m), poor interoperability between different devices, and the high cost of wide-area coverage. They have not yet formed a wide-area coverage sensing system. It is necessary to build a wide-area coverage sensing network based on the communication network and computing network of the communication network.

[0041] As described in the background section, current sensor network architecture designs do not consider the networking issues between sensor networks and non-3GPP sensing devices. Furthermore, sensor networks require diverse and complex sensing processing capabilities to handle different types of multimodal sensing data, making network deployment challenging.

[0042] For example, non-3GPP sensing devices with functions such as cameras, millimeter-wave radar, lidar, sonar, ultra-wideband (UWB), wireless connectivity, and Global Positioning System (GPS) can collect sensing data about target objects or the environment. However, the raw data collected by different non-3GPP sensing devices is multimodal. For example, cameras perceive video results, millimeter-wave radar obtains time-delay Doppler images, and lidar obtains sonar images. Therefore, in the original single-sensor access process, each non-3GPP sensing device requires a complex multimodal sensing data processing process to access the sensing network. This makes multimodal sensing network deployment difficult due to limitations in computing power / capabilities.

[0043] In view of this, embodiments of this disclosure provide a sensing fusion method, which includes: fusing first sensing data and second sensing data based on received control information to obtain fused sensing data. The first sensing data originates from at least one sensing node, and the second sensing data is data that meets the requirements of a sensing network, obtained by preprocessing raw data collected by at least one sensing device. In this way, the raw data collected by the sensing devices is preprocessed to meet the requirements of the sensing network, so that the sensing network does not need to have the processing capability for raw data collected by various sensing devices (e.g., non-3GPP sensing devices) during the sensing data fusion process. This allows the sensing network to concentrate its computing power on network fusion rather than performing basic processing on the raw data collected by each sensing device, enabling flexible networking of the sensing network with a large number of sensing devices and improving the networking and sensing capabilities of the sensing network.

[0044] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0045] The perception fusion method provided in this disclosure can be applied to systems with various communication standards. For example, the systems to which the perception fusion method provided in this disclosure can be applied include, but are not limited to, long term evolution (LTE) systems, various versions based on LTE evolution, 5th generation (5G) communication systems, wireless fidelity (Wi-Fi) systems, 3GPP-related communication systems, ambient internet of things (Ambient IoT) systems, or systems integrating multiple systems. In addition, the perception fusion method provided in this disclosure can also be applied to future-oriented communication systems (such as 6G communication systems), etc., and this disclosure does not limit this application.

[0046] The application scenarios of the sensing fusion method provided in this disclosure include intelligent transportation, low-altitude economy, and water area sensing, which simultaneously have non-3GPP sensing equipment and communication coverage.

[0047] For example, Figure 1 The diagram illustrates the architecture of a communication system according to an embodiment of this disclosure. The communication system includes a sensing network and a non-3GPP sensing network. The sensing network includes a sensing function (SF) in the core network domain, a second computing node and a first node in the access network domain, and sensing nodes. The non-3GPP sensing network includes a first computing node and sensing devices (including non-3GPP sensing devices). Each node or device can be one or more, and the number is not limited.

[0048] In some embodiments, the sensing network element, the second computing node, the first node, and the first computing node are logical nodes. The sensing node and the sensing device are physical nodes. Each node or device may have other names; for example, the first computing node may also be called an edge computing node, the second computing node may be called a sensing computing node, and the first node may also be called a multimodal computing / fusion node. This disclosure does not impose any limitations on these names.

[0049] In some embodiments, sensing network elements support sensing management and control in integrated communication and sensing networks. This includes sensing service generation and distribution, sensing data acquisition and transmission, sensing resource scheduling, sensing processing resource scheduling, and sensing signaling interaction. Here, sensing refers to the acquisition of multi-dimensional information about the physical world through wireless sensing technologies (such as radar sensing, environmental sensing, and human behavior sensing), involving the perception of the physical world such as the environment, user behavior, and scenarios, rather than the perception of computing power and network quality. For example, sensing network elements can be core network equipment rooms, data centers, etc.

[0050] In some embodiments, the sensing network element can allocate computing power and generate corresponding nodes. For example, a first node can be generated to be responsible for collecting sensing data and performing operations such as time alignment, fusion, and correlation.

[0051] In some embodiments, the first node refers to a computing node with multimodal perception-related computing and scheduling functions. Multimodal perception includes integrated sensing and various sensing devices such as cameras, millimeter-wave radar, lidar, and GPS.

[0052] In some embodiments, the second computing node refers to a computing node with dedicated processing capabilities related to synesthesia. The second computing node can process the raw data collected by the synesthesia nodes accessed within the covered sensing area.

[0053] In some embodiments, a sensing node is a node in a communication-sensing integrated network that simultaneously possesses communication and sensing functions. It can transmit and / or receive communication signals, sensing signals, and control information. A sensing node can be a base station, a terminal, customer premises equipment (CPE), a reconfigurable intelligent surface (RIS), or other devices with communication and sensing capabilities.

[0054] In some embodiments, the first node is connected to a second computing node in the corresponding sensing area, or the second computing node and the first node can share the same physical device. The second computing node and / or the first node can be a server, computing board, computing resources of a base station, computing center or other computing device.

[0055] In some embodiments, the first computing node has computing capabilities and can connect to multiple sensing devices. The first computing node can process the raw data collected by the connected sensing devices, and can also process multimodal fusion sensing data. This allows the sensors associated with the sensing devices and their associated computing devices to connect to the sensor network through a defined interface. For example, the first computing node can be a server, an edge computing center, a terminal, a drone, or a device with computing capabilities such as the computing resources of the sensor network.

[0056] In some embodiments, sensing devices include devices that do not have 3GPP communication capabilities but have the ability to acquire target object or environmental sensing data, such as mobile devices with sensing functions or sensors such as cameras, millimeter-wave radar, lidar, sonar, UWB or mobile hotspot (wireless fidelity, WIFI), etc. These sensing devices may also be referred to as non-3GPP sensing devices.

[0057] In some embodiments, the sensing area covered by the first computing node is smaller than the sensing area covered by the second computing node.

[0058] In some embodiments, the first node and the first computing node corresponding to the sensing area of ​​the first node are connected. The connection between the first node and the first computing node can be Ethernet, intra-board communication (communication between two modules on the same circuit board), or wireless communication, etc.

[0059] In this disclosure, the base station can be a base station in Long Term Evolution (LTE), Long Term Evolution Advanced (LTEA), or an evolved Node B (eNB or eNodeB), a base station device in a 5G network, or a base station in a future communication system, etc. The base station can include various macro base stations, micro base stations, home base stations, wireless remote extensions, reconfigurable intelligent surfaces (RISs), routers, relays, transmission and reception points (TRPs), WIFI devices, UEs, and other network-side devices. This disclosure does not limit this type of device in the embodiments.

[0060] In this disclosure, a terminal can be a device with wireless transceiver capabilities. Terminals can be passive devices, ambient IoT devices, mobile phones, tablets, computers with wireless transceiver capabilities, virtual reality (VR) terminals, augmented reality (AR) terminals, wireless terminals in industrial control, self-driving, remote medical care, smart grids, transportation safety, smart cities, smart homes, etc. The embodiments of this disclosure do not limit the application scenarios. A terminal may also be referred to as a user, UE, access terminal, UE unit, UE station, mobile station, mobile station, remote station, remote terminal, mobile device, UE terminal, wireless communication device, UE agent, or UE device, etc., and the embodiments of this disclosure do not limit this to these terms.

[0061] It should be understood that Figure 1 This is an exemplary structural diagram, such as Figure 1 The number of devices included in the illustrated communication system is unlimited; for example, the number of base stations and terminals is unlimited. Furthermore, except... Figure 1 In addition to the equipment shown, Figure 1 The communication system shown may also include other devices, which are not limited thereto.

[0062] This disclosure provides a perception fusion method applied to a first node. For example... Figure 2 As shown, the method includes the following steps:

[0063] S101, The first node receives control information.

[0064] The control information is used to instruct the fusion of the first and second sensing data.

[0065] The first sensing data comes from at least one sensory node.

[0066] The second type of sensing data is data that meets the requirements of a sensory network, obtained by preprocessing the raw data collected by at least one sensing device (such as target detection, signal analysis, unified data format, basic sensing processing, and data standardization). For example, the requirements of a sensory network include a unified data format. Since multimodal sensing devices have different types of sensing capabilities, the formats of the raw data collected by various sensing devices are not uniform. By unifying the data format and fusing it into the sensory network, the need for the sensory network to have multiple complex multimodal sensing processing capabilities is avoided.

[0067] In some embodiments, the control information is provided by the core network; for example, a sensing network element in the core network provides control information to the first node. Additionally, the sensing network element can also use the control information to assign sensing fusion tasks to different nodes. For example, the first node may assign tasks to the sensing network element. Furthermore, the sensing network element can also assign the first computing node and the second computing node, as described below.

[0068] In some embodiments, the sensing device includes a non-3GPP sensing device.

[0069] In some embodiments, the sensing device has at least one of the following sensor types: camera, millimeter-wave radar, lidar, sonar, wireless connectivity, UWB, and GPS. For example, video information from a camera or point cloud information from a millimeter-wave radar can be used as raw data collected by the sensing device.

[0070] In some embodiments, the first node can statistically analyze the sensing data generated by the sensing device over a period of time to determine the sensing area covered by the sensing device. For example, the first node can find the smallest bounding rectangle of all points covered by the sensing device as the sensing area covered by the sensing device.

[0071] It is understandable that the sensing area covered by the sensing device is relatively small, and can only form local sensing coverage. Therefore, the sensing device can also be directly connected to the sensor network, or the sensor network can allocate computing power to generate data that meets the requirements of the sensor network by preprocessing the raw data collected by at least one sensing device by the edge computing nodes, which is the second sensing data.

[0072] In some embodiments, a first computing node is connected to at least one sensing device, and the first computing node has the function of processing raw data collected by the at least one sensing device. The second sensing data is data that meets the requirements of the sensory network, obtained by the first computing node processing the raw data collected by the at least one sensing device. These processing steps are basic sensing processes, such as target detection, signal parsing, or unified data formatting. The data is then fused into the sensory network through a unified interface (preprocessed by the first computing node), avoiding the need for the sensory network to possess multiple complex multimodal sensing processing capabilities.

[0073] In this process, the first computing node can determine the timestamp of the second sensing data at the same time as determining the second sensing data. Further details will not be elaborated upon hereafter.

[0074] In some embodiments, the second sensing data includes at least one of the following: identification information of the sensing object, longitude, latitude and altitude of the sensing object, three-dimensional velocity of the sensing object, three-dimensional acceleration of the sensing object, distance and azimuth of the sensing object, pitch angle of the sensing object, type of the sensing object (e.g., classification of people, vehicles, drones, ships, non-motorized vehicles, etc.), confidence level, and reflected signal power of the sensing object.

[0075] In some embodiments, for each first computing node, the sensing area covered by the first computing node can be determined based on the sensing area covered by at least one sensing device connected to the first computing node. For example, the sensing area covered by the first computing node is the maximum sensing area covered by all sensing devices connected to the first computing node.

[0076] In some embodiments, for each first computing node, statistical analysis is performed on the second sensing data generated by the first computing node over a period of time to determine the sensing area covered by the first computing node. For example, the smallest bounding rectangle of all sensing objects in the second sensing data is found as the sensing area covered by the first computing node.

[0077] In some embodiments, the second computing node is connected to at least one sensing node, and the second computing node has the ability to process the raw data collected by the at least one sensing node to obtain first sensing data. The first sensing data is obtained by the second computing node fusing the raw data collected by the at least one sensing node. Further details will not be elaborated upon hereafter.

[0078] For example, Figure 3 A sensing network setup process is provided, namely, the process by which the second computing node generates and reports the first sensing data, such as... Figure 3 As shown, it includes the following steps:

[0079] a1. The sensing network element determines the sensing configuration information (or control information) based on the sensing requirements.

[0080] The perception configuration information includes at least one of the following: perception area, perception type, allocation of the second computing node and sensor node required to complete the task, and the connection relationship between each node.

[0081] In some embodiments, the sensing area information may include a preset set of coordinate information, and the area to be sensed can be obtained by connecting the preset set of coordinate information. For example, the sensing area information includes a rectangular area composed of four latitude and longitude coordinates.

[0082] In some embodiments, the sensing type includes at least one of the following: target tracking sensing, area monitoring sensing, environment reconstruction sensing, and micro-deformation sensing. Each second computing node is connected to at least one sensing node.

[0083] a2. The sensing network element sends the sensing configuration information to the second computing node.

[0084] a3. The second computing node sends the sensing configuration information to at least one sensing node. Alternatively, the sensing network element sends the sensing configuration information to the sensing node.

[0085] a4. At least one sensing node connects to the second computing node based on the sensing configuration information and reports the sensing data to the second computing node.

[0086] a5. The second computing node merges the sensing data of at least one connected sensing node to form the first sensing data.

[0087] In this disclosure, the first sensing data includes at least one of the following: identification information of the sensing object, longitude, latitude and altitude of the sensing object, three-dimensional velocity of the sensing object, three-dimensional acceleration of the sensing object, distance and azimuth of the sensing object, pitch angle of the sensing object, type of the sensing object (e.g., classification of people, vehicles, drones, ships, non-motorized vehicles, etc.), confidence level of the sensing data corresponding to the sensing object, and reflected signal power of the sensing object.

[0088] a6. The second computing node reports the first sensing data to the sensing network element. When there are multiple second computing nodes, each second computing node can transmit its sensing data to a target second computing node, which then further integrates the data before reporting it to the sensing network element. This disclosure does not limit the connection relationships between the second computing nodes or the number of each node. At this point, the sensory network integration is complete, with each sensory node covering a sensing area.

[0089] It is understandable that when the first node performs multimodal perception fusion, the first node needs to acquire the first perception data, for example, through the perception network element or through the second computing node.

[0090] S102. The first node fuses the first sensing data and the second sensing data based on the control information to obtain fused sensing data.

[0091] In some embodiments, the control information includes fusion method information. Fusing the first sensing data and the second sensing data based on the control information to obtain fused sensing data includes: determining the fusion method based on the fusion method information in the control information; and fusing the first sensing data and the second sensing data based on the fusion method to obtain fused sensing data.

[0092] The fusion method information includes at least one of the following: timestamp alignment method, computing power distribution, and multimodal fusion type.

[0093] Due to the different access situations of non-3GPP sensing devices, the integration methods are also divided into two categories: distributed integration and centralized integration.

[0094] In some embodiments, the fusion method information includes a multimodal fusion type, and the fusion method is determined based on the fusion method information in the control information, including:

[0095] When the multimodal fusion type is distributed fusion, the fusion method is determined to be distributed fusion.

[0096] The multimodal fusion type is centralized fusion, and the fusion method is determined to be centralized fusion.

[0097] In some embodiments, the fusion method information includes a timestamp alignment method. Determining the fusion method based on the fusion method information in the control information includes:

[0098] With the timestamp alignment method aligned to the timestamp of the second sensing data, the fusion method is determined to be distributed fusion.

[0099] The timestamp alignment method is aligned to the timestamp of the first perceived data, and the fusion method is determined to be centralized fusion.

[0100] In some embodiments, the fusion method information includes computing power distribution. Determining the fusion method based on the fusion method information in the control information includes:

[0101] In the case of computing power distribution, the first computing node is responsible for multimodal fusion, and the fusion method is determined to be distributed fusion;

[0102] In terms of computing power distribution, the first node is responsible for multimodal fusion, and the fusion method is determined to be centralized fusion.

[0103] In some embodiments, when the first computing node is responsible for multimodal fusion, the first node is responsible for sensing data integration and forwarding, global multimodal fusion and reporting. When the first node is responsible for multimodal fusion, the first computing node is responsible for sending the second sensing data to the first node.

[0104] In some embodiments, the fusion method is distributed fusion if at least one of the following conditions is met:

[0105] The computing power of the first computing node is greater than or equal to the first threshold (e.g., the first threshold is 10 TOPS);

[0106] The average access time of at least one sensing device is less than the first preset time (e.g., the first preset time is 1 minute, which means that the device frequently joins / leaves, and the sensing scenario is highly dynamic).

[0107] At least one sensing device contains a sensor (such as a camera) that has no speed sensing capability;

[0108] At least one sensing device has a clock accuracy less than the second threshold, for example, at least one sensing device has an error greater than 1ms between the clock of the sensing device and the real clock.

[0109] In some embodiments, the fusion method is centralized fusion if at least one of the following conditions is met:

[0110] The average access duration of at least one sensing device is greater than the second preset duration (e.g., the second preset duration is 1 minute, which means that the device does not frequently join / leave, and the sensing scenario is static or semi-static).

[0111] At least one sensing device contains a sensor that supports velocity / acceleration measurement;

[0112] At least one sensing device has a clock accuracy greater than the third threshold, for example, at least one sensing device has an error of less than 1ms between the clock of the sensing device and the real clock;

[0113] The core network computing power is greater than or equal to the fourth threshold (e.g., the fourth threshold is 100 TOPS).

[0114] In some embodiments, the control information is determined based on the first request information of the first computing node.

[0115] The first request information is used to request the connection of at least one sensing device to the sensor network, and the first computing node has the ability to preprocess the raw data collected by at least one sensing device (such as target detection, signal parsing, unified data format, basic sensing processing, and data standardization processing).

[0116] In some embodiments, the first request information of the first computing node includes at least one of the following:

[0117] The computing power of the first computing node, for example, 10 TOPS;

[0118] The access duration of at least one sensing device, for example, 10 minutes;

[0119] The type of sensor present in at least one sensing device, such as a camera or millimeter-wave radar;

[0120] The clock accuracy of at least one sensing device can be determined by the error between the clock of the sensing device and the real clock.

[0121] The sensing area covered by the first computing node, for example, the sensing area is a region composed of a preset number of latitude and longitude coordinates.

[0122] In some embodiments, when the control information is provided by the sensing network element, the sensing network element can determine the control information based on the first request information of the first computing node and the computing power of the sensing network element itself.

[0123] In some embodiments, when the fusion method is distributed fusion, the first node fuses the first sensing data and the second sensing data based on the fusion method to obtain fused sensing data, including:

[0124] The first node obtains the first sensing data from the second computing node; the second computing node has the ability to process the raw data collected by at least one sensor node to obtain the first sensing data;

[0125] For each first computing node, the first node sends target sensing data from the first sensing data to the first computing node. The target sensing data is the sensing data in the first sensing data that corresponds to the sensing area covered by the first computing node. The first computing node has the ability to process the raw data collected by at least one sensing device to obtain the first sensing data.

[0126] The first node receives the third sensing data sent by each first computing node; wherein, the third sensing data sent by the first computing node is obtained by the first computing node through timestamp alignment and correlation fusion of the target sensing data and the second sensing data processed by the first computing node;

[0127] The first node spatially fuses the third-sensory data from each first computing node to obtain fused sensing data.

[0128] In some embodiments, the third sensing data is obtained by resampling the third sensing data of the first computing node according to a pre-configured sensing refresh rate. Since the sensing refresh rates are different for the sensing areas covered by different sensing devices, pre-configuring the sensing refresh rate avoids the need for complex algorithm design for sensing fusion.

[0129] In some embodiments, the first node receives third sensing data sent by each first computing node, including: the first node receives the third sensing data sent by each first computing node based on a pre-configured sensing refresh rate.

[0130] In some embodiments, when the fusion method is centralized fusion, the first node fuses the first sensing data and the second sensing data based on the fusion method to obtain fused sensing data, including:

[0131] The first node obtains the first sensing data from the second computing node; the second computing node has the ability to process the raw data collected by at least one sensor node to obtain the first sensing data;

[0132] The first node obtains the second sensing data from the first computing node; the first computing node has the ability to process the raw data collected by at least one sensing device to obtain the first sensing data;

[0133] The first node aligns and fuses the first and second perception data using timestamps to obtain fused perception data.

[0134] In some embodiments, the first node resamples the fused sensing data based on a pre-configured sensing refresh rate; the first node then reports the resampled fused sensing data.

[0135] In some embodiments, the pre-configured perceived refresh rate is carried in the control information.

[0136] In some embodiments, association fusion includes at least one of the following:

[0137] The sensing data with the highest reflected signal power among the sensing data corresponding to the same sensing object is taken as the fused data of the same sensing object.

[0138] The sensor data with the highest confidence among the sensor data corresponding to the same sensor object is taken as the fused data of the same sensor object.

[0139] The sensing data corresponding to the same sensing object are weighted and merged based on the reflected signal power to form the fused data of the same sensing object;

[0140] The sensory data corresponding to the same sensory object are weighted and merged based on confidence level to form the fused data of the same sensory object;

[0141] The sensory data corresponding to the same sensory object are weighted and merged based on confidence level to form the fused data of the same sensory object;

[0142] The sensory data corresponding to the same sensory object are combined by weighting the arithmetic average and merging them into the fused data of the same sensory object.

[0143] In some embodiments, at least two sensing objects are considered to be the same sensing object if the difference between sensing data from at least two sensing objects is less than a threshold. For example, sensing data includes latitude and longitude, azimuth, pitch angle, distance, etc. If the difference between these sensing data of the sensing objects is less than the threshold, it indicates that the sensing objects are spatially close, i.e., they may be the same sensing object.

[0144] For example, such as Figure 4 As shown, a perception fusion method is provided, employing a distributed fusion approach. The first computing node is responsible for preprocessing perception data from at least one sensing device and performing multimodal fusion. The first node is also responsible for integrating and forwarding perception data from the sensor nodes, and for globally fusion and forwarding the perception data obtained from the multimodal fusion of the first computing node. Specifically, the method includes the following steps:

[0145] b1. The first computing node receives and stores raw data collected by at least one sensing device connected to it.

[0146] b2. Second sensing data that meets the requirements of a sensory network, obtained by preprocessing the raw data collected by at least one sensing device by the first computing node.

[0147] b3. The first computing node sends the first request information to the sensing network element.

[0148] b4. The sensing network element determines control information based on the first request information and its own computing power. This control information includes fusion method information and a pre-configured sensing refresh rate. The fusion method information in the control information is used to determine if the fusion method is distributed fusion. For example, the fusion method information in the control information may include the timestamp aligned to the second sensing data. Another example is that the distribution configuration in the fusion method information may indicate that the first computing node is responsible for multimodal fusion. Yet another example is that the multimodal fusion type in the fusion method information in the control information may be distributed fusion.

[0149] b5. The sensing network element sends control information to the first node.

[0150] b6. The sensing network element sends control information to the first computing node.

[0151] b7. The first node obtains the first perception data from the second computing node.

[0152] The first sensing data refers to the data obtained after networking and fusing at least one sensing node. The process for acquiring the first sensing data can be referred to the above. Figure 3 As described above, I will not repeat it here.

[0153] b8. The first node sends the target perception data from the first perception data to the first computing node. The target perception data is the perception data in the first perception data that corresponds to the perception area covered by the first computing node.

[0154] b9. The first computing node timestamps the target perception data and the second perception data, and then fuses them to obtain the third perception data.

[0155] In some embodiments, the first computing node extracts the timestamp of the target sensing data. The first computing node aligns the timestamp of the target sensing data to the timestamp of the second sensing data. For example, the first computing node uses the timestamp of the second sensing data as a reference and inserts the target sensing data into the time series of the second sensing data using interpolation or a sliding window algorithm.

[0156] In some embodiments, the first computing node correlates and fuses the perception data of the same perceived object from the target perception data and the second perception data to obtain the third perception data. For example, spatial location matching is used to match data from different sensors. If there are conflicts in the perception data from different sensors (such as inconsistent types of perceived objects), a confidence weighting method (such as prioritizing data with higher signal power) is used for decision-making.

[0157] b10. The first node receives the third sensing data sent by the first computing node.

[0158] In some embodiments, the third sensing data is obtained by resampling the third sensing data of the first computing node according to a pre-configured sensing refresh rate.

[0159] In some embodiments, the first node receives third sensing data sent by the first computing node based on a pre-configured sensing refresh rate.

[0160] In some embodiments, the third sensing data includes at least one of the following: identification information of the sensing object, longitude, latitude and altitude of the sensing object, three-dimensional velocity of the sensing object, three-dimensional acceleration of the sensing object, distance and azimuth of the sensing object, pitch angle of the sensing object, type of the sensing object (e.g., classification of people, vehicles, drones, ships, non-motorized vehicles, etc.), confidence level, and reflected signal power of the sensing object.

[0161] b11. The first node spatially fuses the third sensing data of at least one first computing node to obtain fused sensing data.

[0162] For example, the spatial fusion of third-sensory data from the first computing node by the first node includes the following steps:

[0163] 1) The first node matches and associates the third-sensing data reported by the first computing node: it determines whether the difference between the sensing data originating from at least two sensing objects is less than a threshold. If the difference is less than the threshold, the at least two sensing objects are determined to be the same sensing object. For example, parameters such as the latitude and longitude, azimuth, elevation, and distance of the sensing objects in the sensing data are used to determine whether the sensing data are close in space and time. Since time alignment has already been performed, spatial proximity is considered at the same time. The judgment criteria are: if the difference in latitude and longitude is less than the threshold, or the difference in azimuth / elevation / distance is less than the threshold, then these sensing data are considered to correspond to the same sensing object.

[0164] 2) Merge the location information from the sensing data corresponding to the same sensing object to obtain fused data for the same sensing object. This includes one of the following methods:

[0165] a. Select the location information from the sensing data with the highest reflected signal power; b. Select the location information from the sensing data with the highest confidence level;

[0166] c. Weighted combining based on reflected signal power;

[0167] d. Confidence-weighted merging;

[0168] e. Weighted merging based on arithmetic average, etc.

[0169] Other information in the sensing data of the same object can be complemented, for example, by combining a sensor, millimeter-wave radar, and a camera. The millimeter-wave radar provides the speed information of the object, and the camera provides the type information of the object.

[0170] b12. The first node sends fused sensing data to the sensing network element.

[0171] The above steps ultimately form a perception map that covers the entire area.

[0172] Figure 4 For a more detailed description of b1 to b12, as well as a more detailed description of each technical feature therein, and a description of the beneficial effects, please refer to the description in the above embodiments or examples, which will not be repeated here. Figure 4 The execution order of the steps can be adjusted, some steps can be omitted, or there may be other steps. The specific order is determined based on the actual application scenario and is not restricted.

[0173] For example, such as Figure 5As shown, a perception fusion method is provided, employing a centralized fusion approach. The first computing node is responsible for preprocessing and forwarding perception data from at least one sensing device, and is also responsible for multimodal fusion. Specifically, the method includes the following steps:

[0174] c1. The first computing node receives and stores raw data collected by at least one sensing device connected to it.

[0175] c2. Second sensing data that meets the requirements of a sensory network, obtained by preprocessing the raw data collected by at least one sensing device by the first computing node.

[0176] c3. The first computing node sends the first request information to the sensing network element.

[0177] c4. The sensing network element determines control information based on the first request information and its own computing power. This control information includes fusion method information and a pre-configured sensing refresh rate. The fusion method information in the control information is used to determine if the fusion method is centralized fusion. For example, the fusion method information in the control information may include the timestamp aligned to the first sensing data. Another example is that the distribution information in the control information may include the first node being responsible for multimodal fusion. Yet another example is that the multimodal fusion type in the control information may be centralized fusion.

[0178] c5. The sensing network element sends control information to the first node.

[0179] c6. The sensing network element sends control information to the first computing node.

[0180] c7. The first node obtains the first sensing data from the second computing node. The first sensing data is the data obtained after networking and fusing at least one sensing node.

[0181] c8. The first node obtains the second sensing data from the first computing node.

[0182] c9. The first node aligns the timestamps of the first and second sensing data and fuses them to obtain fused sensing data.

[0183] In some embodiments, the first node extracts the timestamps of the first and second sensing data. The first node aligns the timestamp of the second sensing data to the timestamp of the first sensing data. For example, the first node uses the timestamp of the first sensing data as a reference and inserts the second sensing data into the time series of the first sensing data using interpolation or a sliding window algorithm.

[0184] In some embodiments, the first node associates and fuses the perception data of the same perception object in the first perception data and the second perception data.

[0185] For example, the association and fusion process includes the following steps:

[0186] 1) Determine whether the difference between the sensing data originating from at least two sensing objects in the first and second sensing data is less than a threshold. If the difference is less than the threshold, the at least two sensing objects are determined to be the same sensing object. For example, using parameters such as the latitude and longitude, azimuth, elevation, and distance of the sensing objects in the first and second sensing data, determine whether the sensing data are close in space and time. Since time alignment has already been performed, spatial proximity is considered at the same time. The judgment criterion is that if the difference in latitude and longitude is less than the threshold, or the difference in azimuth / elevation / distance is less than the threshold, then these sensing data are considered to correspond to the same sensing object.

[0187] 2) Merge the location information from the sensing data corresponding to the same sensing object to obtain fused data for the same sensing object. This includes one of the following methods:

[0188] a. Select the location information from the sensing data with the highest reflected signal power;

[0189] b. Select location information from the sensing data with the highest confidence level;

[0190] c. Weighted combining based on reflected signal power;

[0191] d. Confidence-weighted merging;

[0192] e. Weighted merging based on arithmetic average, etc.

[0193] Other information in the sensing data of the same object can be complemented, for example, by combining a sensor, millimeter-wave radar, and a camera. The millimeter-wave radar provides the speed information of the object, and the camera provides the type information of the object.

[0194] Understandably, since the first perception data is inherently global, the result of the fusion is the result of global fusion.

[0195] c10. The first node sends fused sensing data to the sensing network element.

[0196] In some embodiments, the first node resamples the fused sensing data based on a pre-configured sensing refresh rate; the first node sends the sampled fused sensing data to the sensing network element.

[0197] In some embodiments, the fused sensing data includes at least one of the following: identification information of the sensing object, longitude, latitude and altitude of the sensing object, three-dimensional velocity of the sensing object, three-dimensional acceleration of the sensing object, distance and azimuth of the sensing object, pitch angle of the sensing object, type of the sensing object (e.g., classification of people, vehicles, drones, ships, non-motorized vehicles, etc.), confidence level, and reflected signal power of the sensing object.

[0198] The above steps ultimately form a perception map that covers the entire area.

[0199] Figure 5 For a more detailed description of C1 to C10, as well as a more detailed description of each technical feature therein, and a description of the beneficial effects, please refer to the description in the above embodiments or examples, which will not be repeated here. Figure 5 The execution order of the steps can be adjusted, some steps can be omitted, or there may be other steps. The specific order is determined based on the actual application scenario and is not restricted.

[0200] Understandably, distributed fusion is suitable for scenarios where non-3GPP sensing devices have low clock accuracy and lack speed sensing data for accurate time interpolation. It ensures fusion reliability through time interpolation of data collected by sensing nodes and inherently supports distributed computing architecture, facilitating full utilization of computing power. Furthermore, it adapts to the fusion needs of highly dynamic network scenarios, such as in vehicle-to-everything (V2X) scenarios where vehicle access / loss is frequent. Distributed fusion can fully utilize vehicle computing power and avoid fusion failure due to the loss of some sensing data.

[0201] Centralized fusion can be applied to scenarios where non-3GPP sensing devices have speed sensing data and can perform predictive interpolation. It achieves efficient one-time global fusion with the sensing network as the main body, and is suitable for high dynamic scenarios such as vehicle-to-everything (V2X) and drone swarms.

[0202] Understandably, the fusion method can be switched. For example, if the computing power of the first computing node starts to fall below the first threshold after selecting distributed fusion, it can be switched to centralized fusion. Or, if network dynamics change after selecting centralized fusion (i.e., the average access time of at least one sensing device is found to be less than the first preset time), it can be switched to distributed fusion.

[0203] The method provided in this disclosure will be described below in conjunction with specific application scenarios.

[0204] Scenario 1: Non-3GPP sensing networks have edge computing nodes and adopt distributed fusion.

[0205] For example, such as Figure 6As shown, consider a traffic monitoring scenario where cameras at traffic intersections and edge computing nodes form a local non-3GPP sensing network. Furthermore, by adding integrated sensing functionality to the existing communication coverage, a wide-area sensing network is formed. One sensing computing node is responsible for performing sensing calculations and reporting the sensing data from at least one sensing node. Sensing network elements are connected to multimodal computing nodes, which in turn are connected to both the sensing computing node and the edge computing node.

[0206] The flexible networking scheme for non-3GPP sensing devices and sensing networks is designed as follows:

[0207] d1. Generation of control information for sensor network (sensing network element):

[0208] Based on the traffic monitoring requirements, the sensing network element determines the sensing area as the intersection area and the sensing type as vehicle sensing. It then allocates corresponding sensory computing nodes and sensory nodes, along with their connection relationships, according to the sensing area and type. The corresponding connection relationship information (via control information) is then sent to the sensory computing nodes and sensory nodes.

[0209] d2. Synesthesia Network Integration (Synesthesia Computing Nodes):

[0210] A sensing node connects to a sensing computing node. The sensing computing node then merges the sensing data from at least one of its connected sensing nodes to form the first sensing data. The format of the first sensing data is as follows: identification information of the sensing object (e.g., a vehicle), longitude, latitude, and altitude of the sensing object, three-dimensional velocity of the sensing object, three-dimensional acceleration of the sensing object, distance and azimuth angle of the sensing object, pitch angle of the sensing object, type of the sensing object, confidence level, and reflected signal power of the sensing object.

[0211] The sensing node can report the first sensing data to the sensing network element.

[0212] At this point, the integration of the sensor network is complete, with each sensor node covering a sensing area.

[0213] d3. Connection and sensing of non-3GPP sensing devices (edge ​​computing nodes):

[0214] 1) Two cameras are connected to the edge computing node. The two cameras collect raw data (such as video information) and send the raw data to the edge computing node.

[0215] 2) Edge computing nodes perform basic target detection and perception processing on the raw data collected by the cameras, generating non-3GPP perception results. These results may include, for example, the vehicle ID, vehicle category, timestamp of the perception data, and the vehicle's GPS coordinates.

[0216] 3) Edge computing nodes encapsulate non-3GPP sensing results into a unified data format to obtain second sensing data. For example, the format is: identification information of the sensing object (such as a vehicle), longitude, latitude and altitude of the sensing object, three-dimensional velocity of the sensing object, three-dimensional acceleration of the sensing object, range and azimuth of the sensing object, pitch angle of the sensing object, type of the sensing object, confidence level, and reflected signal power of the sensing object.

[0217] 4) Statistically analyze the second sensing data generated over a period of time to determine the sensing area covered by the edge computing node. For example, find the smallest bounding rectangle of all sensing objects included in the second sensing data as the sensing area covered by the edge computing node. Alternatively, use the maximum sensing coverage of two cameras as the sensing area covered by the edge computing node, where the maximum sensing coverage of each camera is based on the smallest bounding rectangle of all points that each camera can cover.

[0218] d4. The edge computing node sends a first request message to the sensing network element, including at least one of the following:

[0219] A. The computing power of edge computing nodes, such as 10 TOPS;

[0220] B. The access duration of at least one sensing device, for example, 30 minutes;

[0221] C. At least one sensing device with a sensor type, such as a camera;

[0222] D. The clock accuracy of at least one sensing device, for example, 1ms error;

[0223] E. The sensing area covered by edge computing nodes, such as a rectangular area consisting of four latitude and longitude coordinates.

[0224] d5. Assigning multimodal sensing fusion tasks to sensing network elements:

[0225] 1) Perception network elements are allocated computing capabilities to generate multimodal computing nodes. After collecting data, the multimodal computing nodes perform time alignment, correlation and fusion and other operations.

[0226] 2) The sensing network elements and multimodal computing nodes are connected to the synesthetic computing nodes and edge computing nodes of the corresponding sensing areas, respectively.

[0227] d6. The sensing network element determines that the camera sensing lacks speed information, the clock accuracy is low, and the edge nodes have sufficient computing power. Therefore, the sensing network element adopts distributed fusion, allocating computing power to generate multimodal computing nodes, which are then connected to the corresponding sensing area's sensory computing nodes and edge computing nodes.

[0228] d7. The sensing network element sends control information, including the multimodal fusion type being distributed fusion, the sensing refresh rate being 1Hz, and the time alignment method being aligned to the timestamp of the camera.

[0229] d8. Sensing data integration and forwarding (multimodal computing node):

[0230] 1) The multimodal computing node receives control information from the sensing network element and clarifies that the multimodal fusion type is distributed fusion.

[0231] 2) The multimodal computing nodes store the first sensing data after network fusion from the synesthetic computing nodes. For each edge computing node, the target sensing data in the first sensing data is sent to the edge computing node. The target sensing data is the sensing data in the first sensing data that corresponds to the sensing area covered by the edge computing node.

[0232] d9. Timestamp Alignment and Single Sensor Fusion (Edge Computing Nodes):

[0233] 1) The edge computing node receives sensing data from two cameras and preprocesses it to obtain second sensing data that meets the requirements of a sensory network. The edge computing node also receives target sensing data forwarded by the multimodal computing node. The edge computing node extracts the timestamps of each data packet.

[0234] 2) Edge computing nodes align and fuse the target perception data and the second perception data using timestamps to obtain the third perception data:

[0235] 2a) The edge computing node aligns the target sensing data to the time series of the second sensing data based on the timestamp alignment method set by the sensing network element and the timestamp of the second sensing data, using interpolation or sliding window algorithms.

[0236] 2b) Edge computing nodes associate and fuse the perception data of the same perception object in the target perception data and the second perception data to form the third perception data.

[0237] For example, spatial location can be used to match data from different sensors. If there are conflicts in the sensing data from different sensors (such as inconsistent types of sensed objects), a confidence-weighted approach (such as prioritizing the data with the highest signal power) can be used for decision-making.

[0238] For example, parameters such as the latitude and longitude, azimuth, elevation, and distance of the sensing object can be used to determine whether the sensing data of the sensing objects are close in space and time. Since time alignment has already been performed, spatial proximity is considered at the same time. The judgment criteria are: if the difference in latitude and longitude is less than a threshold, or the difference in azimuth / elevation / distance is less than a threshold, then the sensing data of these sensing objects are considered to correspond to the same sensing object.

[0239] Location information from sensing data corresponding to the same sensing object is fused to form fused data for that object. This can be achieved using one of the following methods: a. selecting location information from the sensing data with the highest reflected signal power; b. selecting location information from the sensing data with the highest confidence level; c. weighted merging based on reflected signal power; d. weighted merging based on confidence level; e. weighted merging based on arithmetic mean, etc. Other information from the sensing data of the same sensing object can be complemented, for example, using a combination of sensors, millimeter-wave radar, and a camera. The millimeter-wave radar provides velocity information for the sensing object, while the camera provides type information.

[0240] 4) Edge computing nodes resample the third-party sensing data according to the pre-configured sensing refresh rate to ensure that it is consistent with the required sensing refresh rate.

[0241] 5) Edge computing nodes report third-party sensing data according to the set sensing refresh rate. The third-party sensing data includes at least one of the following: identification information of the sensing object (such as a vehicle), longitude, latitude and altitude of the sensing object, three-dimensional velocity of the sensing object, three-dimensional acceleration of the sensing object, distance and azimuth of the vehicle, pitch angle of the sensing object, type of the sensing object, confidence level, and reflected signal power of the sensing object.

[0242] d10. Global Multimodal Fusion and Reporting (Multimodal Computing Node):

[0243] The multimodal computing node spatially fuses the third-party sensing data reported by the edge computing node to obtain fused sensing data.

[0244] For example, spatial integration includes the following steps:

[0245] 1) By matching the latitude, longitude, or azimuth in the third sensing data with the sensing results of different regions, it can be determined whether the data of each sensing object in the third sensing data can be spatially correlated.

[0246] 2) Perform trajectory prediction complementarity or error correction on data of the same sensing object.

[0247] 3) The fused sensing data is reported from the multimodal computing nodes to the sensing network elements, and finally forms a sensing map covering the entire domain.

[0248] For example, spatial integration includes the following steps:

[0249] 1) Multimodal computing nodes match and associate the third-party sensing data reported by edge computing nodes: They determine whether the difference between sensing data originating from at least two sensing objects is less than a threshold. If the difference is less than the threshold, the at least two sensing objects are considered to be the same sensing object. For example, parameters such as the latitude and longitude, azimuth, elevation, and distance of the sensing objects in the sensing data are used to determine whether the sensing data are spatially and temporally close. Since time alignment has already been performed, spatial proximity is considered at the same time. The judgment criteria are: if the difference in latitude and longitude is less than the threshold, or the difference in azimuth / elevation / distance is less than the threshold, then these sensing data are considered to correspond to the same sensing object.

[0250] 2) The location information from the sensing data corresponding to the same sensing object is fused to form the fused data for the same sensing object. This includes one of the following methods: a. Selecting the location information from the sensing data with the highest reflected signal power; b. Selecting the location information from the sensing data with the highest confidence level; c. Weighted merging based on reflected signal power; d. Weighted merging based on confidence level; e. Weighted merging based on arithmetic mean, etc. For other information in the sensing data of the same sensing object, data complementarity can be achieved, for example, using a combination of sensor + millimeter-wave radar + camera. The millimeter-wave radar provides the velocity information of the sensing object, and the camera provides the type information of the sensing object.

[0251] 3) The fused sensing data is reported from the multimodal computing nodes to the sensing network elements, and finally forms a sensing map covering the entire domain.

[0252] Scenario 2: Non-3GPP sensing networks lack edge computing nodes and employ a centralized fusion approach.

[0253] For example, such as Figure 7 As shown, consider a traffic monitoring scenario and a vehicle perception scenario. The vehicle has its own GPS sensor, and the vehicle's GPS sensor, together with edge computing nodes, forms a local non-3GPP sensing network. Furthermore, based on the existing communication coverage, after the sensing nodes add integrated sensing functionality, a wide-area sensing network is formed. One sensing computing node is responsible for performing sensing calculations and reporting the sensing data from at least one sensing node. Sensing network elements are connected to multimodal computing nodes, which in turn are connected to both the sensing computing nodes and the edge computing nodes.

[0254] The flexible networking scheme for non-3GPP sensing devices and sensing networks is designed as follows:

[0255] e1. Generation of control information for sensor network (sensing network element):

[0256] The sensing network element determines the sensing area as the road area and the sensing type as moving object sensing based on the vehicle's sensing needs. It then allocates corresponding synesthetic computing nodes and synesthetic nodes, along with their connections, based on the sensing area and sensing type. The corresponding connection information (via control information) is then sent to the synesthetic computing nodes and synesthetic nodes.

[0257] e2. Synesthesia Network Integration (Synesthesia Computing Nodes):

[0258] The synesthetic node connects to the synesthetic computing node, which can network and fuse the sensing data of multiple connected synesthetic nodes to form the first sensing data.

[0259] The sensor node reports the first sensing data to the sensing network element.

[0260] At this point, the integration of the sensor network is complete, with each sensor node covering a sensing area.

[0261] e3. Connection and sensing of non-3GPP sensing devices (edge ​​computing nodes):

[0262] 1) The vehicle's GPS is connected to the sensor network. Since there is no edge computing node, the sensing network element allocates an edge computing node from the computing power on the access network side, and the vehicle sends its own GPS information to the edge computing node.

[0263] 2) Edge computing nodes complete basic target detection and perception processing, generating non-3GPP perception results. These results may include, for example, the identification information of the perceived object (such as a vehicle), the category of the perceived object, the timestamp of the perception data, and the GPS coordinates of the perceived object.

[0264] 3) Edge computing nodes encapsulate non-3GPP sensing results into a unified data format to obtain second sensing data. For example, the format is: identification information of the sensing object (such as a vehicle), longitude, latitude and altitude of the sensing object, three-dimensional velocity of the sensing object, three-dimensional acceleration of the sensing object, range and azimuth of the sensing object, pitch angle of the sensing object, type of the sensing object, confidence level, and reflected signal power of the sensing object.

[0265] 4) Statistically analyze the second sensing data generated over a period of time to determine the sensing area covered by the edge computing nodes. For example, find the smallest bounding rectangle of all sensing objects included in the second sensing data as the sensing area covered by the edge computing nodes.

[0266] e4. The edge computing node sends a first request message to the sensing network element, including at least one of the following:

[0267] A. The computing power of edge computing nodes, such as 10 TOPS, but it is to borrow the computing power of the perceptron network;

[0268] B. The access duration of at least one sensing device, for example, 1 minute;

[0269] C. At least one sensing device with a sensor type, such as GPS;

[0270] D. The clock accuracy of at least one sensing device, for example, an error of 0.1ms;

[0271] E. The sensing area covered by edge computing nodes, such as a rectangular area formed by four latitude and longitude coordinates.

[0272] e5. Assigning multimodal sensing fusion tasks to sensing network elements:

[0273] 1) Perception network elements allocate computing capabilities to generate multimodal computing nodes. After collecting data, the multimodal computing nodes perform time alignment, fusion and correlation operations.

[0274] 2) The sensing network elements and multimodal computing nodes are connected to the synesthetic computing nodes and edge computing nodes of the corresponding sensing areas, respectively.

[0275] e6. The sensing network element determines that GPS sensing provides speed information, has high clock accuracy, and that edge nodes lack sufficient computing power. The sensing network element is determined using centralized fusion, with the sensing network element allocating computing power to generate multimodal computing nodes, which are then connected to the corresponding sensing area's sensory computing nodes and edge computing nodes.

[0276] e7. The sensing network element issues control information, including that the multimodal fusion type is centralized fusion, the sensing refresh rate is 1Hz, and the timestamp alignment method is aligned to the synaptic timestamp (i.e., the timestamp of the first sensing data). The sensing network element allocates computing resources. In the centralized fusion strategy, edge computing nodes are responsible for reporting sensing information, and multimodal computing nodes are responsible for overall fusion sensing.

[0277] e8. Global Multimodal Fusion and Reporting (Multimodal Computing Nodes):

[0278] 1) The multimodal computing node receives the second sensing data and the first sensing data from the edge computing node, and extracts the timestamps of each data packet. The second sensing data is data that meets the requirements of a sensory network, obtained by the edge computing node through preprocessing the vehicle GPS sensing data.

[0279] 2) The multimodal computing node performs timestamp alignment and correlation fusion on the second and first sensing data to obtain fused sensing data:

[0280] 2a) Based on the timestamp alignment method set by the sensing network element, the multimodal computing node aligns the second sensing data to the time series of the first sensing data using interpolation or sliding window algorithms, with the timestamp of the first sensing data as the reference.

[0281] 2b) The multimodal computing node correlates and fuses the sensory data of the same sensory object in the second sensory data and the first sensory data as fused sensory data. Since synesthetic data is naturally global, the result of fusion is the result of global fusion.

[0282] For example, using parameters such as the latitude and longitude, azimuth, pitch angle, and distance of the sensing object, we can determine whether the sensing data of the sensing object (such as a vehicle) are close in space and time. Since time alignment has already been performed, we consider spatial proximity at the same time. The judgment criteria are: if the difference in latitude and longitude is less than a threshold, or the difference in azimuth / pitch angle / distance is less than a threshold, then the sensing data of these sensing objects are considered to correspond to the same sensing object.

[0283] The location information from the sensing data corresponding to the same sensing object is fused together to form the fused data for the same sensing object. This specifically includes one of the following methods:

[0284] a. Select the location information from the sensing data with the highest reflected signal power;

[0285] b. Select location information from the sensing data with the highest confidence level;

[0286] c. Weighted combining based on reflected signal power;

[0287] d. Confidence-weighted merging;

[0288] e. Weighted merging based on arithmetic average, etc.

[0289] Other information in the sensing data of the same object can be complemented, for example, by combining a sensor, millimeter-wave radar, and a camera. The millimeter-wave radar provides the speed information of the object, and the camera provides the type information of the object.

[0290] 4) The multimodal computing nodes resample the fused sensing data according to the pre-configured sensing refresh rate to ensure consistency with the required sensing refresh rate. The multimodal computing nodes report the fused sensing data to the sensing network elements, ultimately forming a sensing map with full coverage.

[0291] Scenario 3: Non-3GPP sensing networks have edge computing nodes, and after adopting a centralized fusion approach, they switch to distributed fusion due to insufficient computing power.

[0292] For example, such as Figure 8As shown, considering a low-altitude economic scenario, millimeter-wave radar from radar equipment senses the low-altitude environment. The radar equipment and edge computing nodes form a local non-3GPP sensing network. Furthermore, based on the existing communication coverage, after adding integrated sensing functionality to the sensing nodes, a wide-area sensing network is formed. One sensing computing node is responsible for performing sensing calculations and reporting the sensing data from at least one sensing node. Sensing network elements are connected to multimodal computing nodes, which in turn are connected to both the sensing computing nodes and the edge computing nodes.

[0293] The flexible networking scheme for non-3GPP sensing devices and sensing networks is designed as follows:

[0294] f1. Generation of control information for sensor network (sensing network element):

[0295] The sensing network element determines the sensing area as the low-altitude economic demonstration area based on the low-altitude economic sensing requirements, and the sensing type as moving object sensing. It then allocates corresponding synesthetic computing nodes and synesthetic nodes, along with their connections, based on the sensing area and type. The corresponding connection information (via control information) is then sent to the synesthetic computing nodes and synesthetic nodes.

[0296] f2. Synesthesia Network Fusion (Synesthesia Computing Nodes):

[0297] The synesthetic node connects to the synesthetic computing node, which can network and fuse the sensing data of multiple connected synesthetic nodes to form the first sensing data.

[0298] At this point, the integration of the sensor network is complete, with each sensor node covering a sensing area.

[0299] f3. Connection and sensing of non-3GPP sensing devices (edge ​​computing nodes):

[0300] 1) Radar equipment collects raw data (such as point cloud information of the radar equipment) and sends the collected raw data to the edge computing node.

[0301] 2) Edge computing nodes complete basic target detection and perception processing, generating non-3GPP perception results. These include, for example, the identifier of the perceived object (such as a drone), the category of the perceived object, the timestamp of the perception data, and the GPS coordinates of the perceived object.

[0302] 3) Edge computing nodes encapsulate non-3GPP sensing results into a unified data format to obtain second sensing data. For example, the format is: identification information of the sensing object (such as a drone), longitude, latitude and altitude of the sensing object, three-dimensional velocity of the sensing object, three-dimensional acceleration of the sensing object, range and azimuth of the sensing object, pitch angle of the sensing object, type of the sensing object (e.g., classification of people, vehicles, drones, ships, non-motorized vehicles, etc.), confidence level, and reflected signal power of the sensing object.

[0303] 4) Statistically determine the sensing area covered by the edge computing nodes by analyzing the second sensing results generated over a period of time. For example, find the smallest bounding rectangle of all sensing objects included in the second sensing data as the sensing area covered by the edge computing nodes.

[0304] f4. The edge computing node sends a first request message to the sensing network element, including:

[0305] A. The computing power of edge computing nodes, such as 10 TOPS;

[0306] B. The access duration of at least one sensing device, for example, 120 minutes;

[0307] C. At least one sensing device with a sensor type, such as millimeter-wave radar;

[0308] D. The clock accuracy of at least one sensing device, for example, an error of 0.1ms;

[0309] E. The sensing area covered by edge computing nodes, such as a rectangular area consisting of four latitude and longitude coordinates.

[0310] f5. Assigning multimodal sensing fusion tasks to sensing network elements:

[0311] 1) Perception network elements allocate computing capabilities to generate multimodal computing nodes. After collecting data, the multimodal computing nodes perform time alignment, fusion and correlation operations.

[0312] 2) The sensing network elements and multimodal computing nodes are connected to the synesthetic computing nodes and edge computing nodes of the corresponding sensing areas, respectively.

[0313] f6. The sensing network element determines that the radar sensing has speed information, high clock accuracy, and sufficient computing power at the edge nodes. The sensing network element adopts a centralized multimodal sensing fusion task, and the sensing network element allocates computing power to generate multimodal computing nodes, which are connected to the corresponding sensing area's sensory computing nodes and edge computing nodes.

[0314] f7. The sensing network element sends control information, including the multimodal fusion type being centralized fusion, the sensing refresh rate being 1Hz, and the time alignment method being aligned to the synesthetic timestamp (i.e., the timestamp of the first sensing data).

[0315] f8. Sensing Data Integration and Forwarding (Multimodal Computing Node):

[0316] 1) The multimodal computing node receives control information from the sensing network element and determines that the fusion mode is a centralized fusion mode.

[0317] 2) Allocate computing resources to sensing network elements. In the centralized fusion strategy, edge computing nodes are responsible for reporting sensing information, while multimodal computing nodes are responsible for overall fusion sensing.

[0318] f9. Global Multimodal Fusion and Reporting (Multimodal Computing Node):

[0319] 1) The multimodal computing node receives second sensing data from the edge node and first sensing data from the sensory computing node, and extracts the timestamps of each data packet. The second sensing data is data that meets the requirements of the sensory network, obtained by the edge computing node through preprocessing the millimeter-wave radar data from the radar device.

[0320] 2) After the multimodal computing node attempts to fuse the second and first sensing data, it encounters insufficient computing power and reports this to the sensing network element.

[0321] f10. The sensing network element receives feedback from the multimodal computing node that the computing power is insufficient, and reissues the control information, including that the multimodal fusion type is distributed fusion, the sensing refresh rate is 1Hz, and the time alignment method is to align to the timestamp of the second sensing data.

[0322] f11. Sensing Data Integration and Forwarding (Multimodal Computing Node):

[0323] 1) The multimodal computing node receives control information from the sensing network element and determines that the fusion mode is a distributed fusion mode.

[0324] 2) The multimodal computing nodes store the first sensing data after network fusion from the synesthetic computing nodes. For each edge computing node, the target sensing data in the first sensing data is sent to the edge computing node. The target sensing data is the sensing data in the first sensing data that corresponds to the sensing area covered by the edge computing node.

[0325] f12. Timestamp Alignment and Single Sensor Fusion (Edge Computing Nodes):

[0326] 1) Edge computing nodes receive sensing data from millimeter-wave radar and preprocess it to obtain second sensing data. Edge computing nodes also receive target sensing data forwarded by multimodal computing nodes. Edge computing nodes extract the timestamps of each data packet.

[0327] 2) Edge computing nodes align and fuse the target perception data and the second perception data using timestamps to obtain the third perception data:

[0328] 2a) The edge computing node aligns the target sensing data to the time series of the second sensing data based on the timestamp alignment method set by the sensing network element and the timestamp of the second sensing data, using interpolation or sliding window algorithms.

[0329] 3) Edge computing nodes associate and fuse the perception data of the same perception object in the target perception data and the second perception data to form the third perception data.

[0330] For example, spatial location can be used to match data from different sensors. If there are conflicts in the sensing data from different sensors (such as inconsistent types of sensed objects), a confidence-weighted approach (such as prioritizing the data with the highest signal power) can be used for decision-making.

[0331] For example, parameters such as the latitude and longitude, azimuth, elevation, and distance of the sensing object can be used to determine whether the sensing data of the sensing objects are close in space and time. Since time alignment has already been performed, spatial proximity is considered at the same time. The judgment criteria are: if the difference in latitude and longitude is less than a threshold, or the difference in azimuth / elevation / distance is less than a threshold, then the sensing data of these sensing objects are considered to correspond to the same sensing object.

[0332] Location information from sensing data corresponding to the same sensing object is fused to form fused data for that object. This can be achieved using one of the following methods: a. selecting location information from the sensing data with the highest reflected signal power; b. selecting location information from the sensing data with the highest confidence level; c. weighted merging based on reflected signal power; d. weighted merging based on confidence level; e. weighted merging based on arithmetic mean, etc. Other information from the sensing data of the same sensing object can be complemented, for example, using a combination of sensors, millimeter-wave radar, and a camera. The millimeter-wave radar provides velocity information for the sensing object, while the camera provides type information.

[0333] 4) Edge computing nodes resample the third-party sensing data according to the pre-configured sensing refresh rate to ensure that it is consistent with the required sensing refresh rate.

[0334] 5) Edge computing nodes report third-party sensing data according to the set sensing refresh rate. The third-party sensing data includes at least one of the following: the identification information of the sensing object, the longitude, latitude and altitude of the sensing object, the three-dimensional velocity of the sensing object, the three-dimensional acceleration of the sensing object, the distance and azimuth of the sensing object, the pitch angle of the sensing object, the type of the sensing object (e.g., classification of people, vehicles, drones, ships, non-motorized vehicles, etc.), confidence level, and the reflected signal power of the sensing object.

[0335] f13. Global Multimodal Fusion and Reporting (Multimodal Computing Node):

[0336] Spatially fuse the third-sensing data reported by edge computing nodes to obtain fused sensing data:

[0337] For example, spatial integration includes the following steps:

[0338] 1) Multimodal computing nodes match and associate the third-party sensing data reported by edge computing nodes: They determine whether the difference between sensing data originating from at least two sensing objects is less than a threshold. If the difference is less than the threshold, the at least two sensing objects are considered to be the same sensing object. For example, parameters such as the latitude and longitude, azimuth, elevation, and distance of the sensing objects in the sensing data are used to determine whether the sensing data are spatially and temporally close. Since time alignment has already been performed, spatial proximity is considered at the same time. The judgment criteria are: if the difference in latitude and longitude is less than the threshold, or the difference in azimuth / elevation / distance is less than the threshold, then these sensing data are considered to correspond to the same sensing object.

[0339] 2) The location information from the sensing data corresponding to the same sensing object is fused to form the fused data for the same sensing object. This includes one of the following methods: a. Selecting the location information from the sensing data with the highest reflected signal power; b. Selecting the location information from the sensing data with the highest confidence level; c. Weighted merging based on reflected signal power; d. Weighted merging based on confidence level; e. Weighted merging based on arithmetic mean, etc. For other information in the sensing data of the same sensing object, data complementarity can be achieved, for example, using a combination of sensor + millimeter-wave radar + camera. The millimeter-wave radar provides the velocity information of the sensing object, and the camera provides the type information of the sensing object.

[0340] 3) The fused sensing data is reported from the multimodal computing nodes to the sensing network elements, and finally forms a sensing map covering the entire domain.

[0341] This disclosure provides a perceptual fusion method applied to a first computing node. For example... Figure 9 As shown, the method includes the following steps:

[0342] S201, The first computing node receives control information.

[0343] The control information is used to instruct the fusion of the first sensing data and the second sensing data; wherein the first sensing data comes from at least one sensing node, and the second sensing data is data that meets the requirements of the sensing network, which is obtained by the first computing node through preprocessing of the raw data collected by at least one sensing device.

[0344] S202, The first computing node performs a perception fusion operation based on control information.

[0345] In some embodiments, the control information includes fusion method information, and the first computing node performs a perceptual fusion operation based on the control information, including:

[0346] The first computing node determines the fusion method based on the fusion method information in the control information, wherein the fusion method information includes at least one of the following: timestamp alignment method, computing power distribution, and multimodal fusion type;

[0347] The first computing node performs a perception fusion operation based on the fusion method.

[0348] In some embodiments, when the fusion method is distributed fusion, performing a perception fusion operation based on the fusion method includes:

[0349] The first computing node receives target sensing data from the first sensing data obtained from the second computing node, which is sent by the first node; the target sensing data is the sensing data in the first sensing data that corresponds to the sensing area covered by the first computing node; the first computing node has the ability to process the raw data collected by at least one sensing device to obtain the second sensing data; the second computing node has the ability to process the raw data collected by at least one sensor node to obtain the first sensing data.

[0350] The first computing node aligns the target perception data and the second perception data with timestamps, and then fuses them to obtain the third perception data.

[0351] The first computing node sends the third sensing data to the first node.

[0352] The second sensing data is obtained by preprocessing the raw data collected by the first computing node from at least one sensing device.

[0353] In some embodiments, the third sensing data is obtained by resampling the third sensing data of the first computing node according to a pre-configured sensing refresh rate.

[0354] In some embodiments, the first computing node sends third sensing data to the first node, including: the first computing node sending third sensing data to the first node based on a pre-configured sensing refresh rate.

[0355] In some embodiments, when the fusion method is centralized fusion, the first computing node performs a perception fusion operation based on the fusion method, including: the first computing node sending second perception data to the first node.

[0356] For a more detailed description of S201 and S202 above, as well as a more detailed description of each technical feature therein, and a description of the beneficial effects, please refer to the corresponding method embodiment section above, which will not be repeated here.

[0357] Based on this, the sensing fusion method provided in this disclosure helps to realize a complete cross-modal wide-area sensing network. This disclosure connects sensors and related computing devices from non-3GPP sensing devices to the sensing network through a specified interface. A complete network fusion interaction process is designed, realizing flexible fusion node selection and fusion procedures, ensuring full utilization of computing nodes and good fusion results. Furthermore, a complete resource allocation and reporting process is designed, including time alignment processing, fusion computing, computing node selection, and reporting node selection.

[0358] Furthermore, the sensing fusion method provided in this disclosure enables flexible networking of the sensing network and non-3GPP sensing devices, allowing non-3GPP sensing devices to be flexibly integrated into the sensing network. This not only enhances the networking capabilities, sensing capabilities (sensing accuracy, sensing range, etc.), and computing power of the sensing network, but also enables the fusion of various non-3GPP sensing devices through the sensing network, realizing a complete cross-modal wide-area sensing network.

[0359] The foregoing primarily describes the solutions of the embodiments of this disclosure from a methodological perspective. The following also illustrates a communication device for executing the perception fusion method in any of the above embodiments and their possible implementations. It is understood that, in order to implement the perception fusion method, the communication device includes hardware structures and / or software modules corresponding to the execution of various functions; those skilled in the art should readily recognize that, in conjunction with the algorithm steps of the examples described in the embodiments of this disclosure, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0360] This disclosure embodiment can divide the communication device into functional modules according to the above method embodiment. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one functional module. The integrated module can be implemented in hardware or software. It should be noted that the module division in this disclosure embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods. The following description uses the example of dividing each functional module according to each function.

[0361] Figure 10 This is a schematic diagram of a communication device provided in an embodiment of the present disclosure. The communication device is applied to a first node. The communication device 70 includes a communication module 71 and a processing module 72.

[0362] The communication module 71 is used to receive control information, which is used to instruct the fusion of the first sensing data and the second sensing data. The first sensing data comes from at least one sensing node, and the second sensing data is data that meets the requirements of the sensing network, obtained by preprocessing the raw data collected by at least one sensing device.

[0363] The processing module 72 is used to fuse the first sensing data and the second sensing data based on the control information to obtain fused sensing data.

[0364] In some embodiments, the control information includes fusion mode information, and the processing module 72 is specifically used for:

[0365] The fusion method is determined based on the fusion method information in the control information, wherein the fusion method information includes at least one of the following: timestamp alignment method, computing power distribution, and multimodal fusion type;

[0366] The first and second sensing data are fused together using a fusion method to obtain fused sensing data.

[0367] In some embodiments, the fusion method information includes a timestamp alignment method, and the processing module 72 is specifically used for:

[0368] With the timestamp alignment method aligned to the timestamp of the second sensing data, the fusion method is determined to be distributed fusion.

[0369] The timestamp alignment method is aligned to the timestamp of the first perceived data, and the fusion method is determined to be centralized fusion.

[0370] In some embodiments, the fusion method information includes computing power distribution, and the processing module 72 is specifically used for:

[0371] In the computing power distribution scenario, the first computing node is responsible for multimodal fusion, and the fusion method is determined to be distributed fusion; the first computing node has the ability to process the raw data collected by at least one sensing device to obtain the second sensing data;

[0372] In terms of computing power distribution, the first node is responsible for multimodal fusion, and the fusion method is determined to be centralized fusion.

[0373] In some embodiments, the fusion method is distributed fusion if at least one of the following conditions is met:

[0374] The computing power of the first computing node is greater than or equal to the first threshold; the first computing node has the ability to process the raw data collected by at least one sensing device to obtain the second sensing data;

[0375] The average access time of at least one sensing device is less than a first preset time.

[0376] At least one sensing device contains a sensor that lacks speed sensing capability;

[0377] At least one sensing device has a clock accuracy less than the second threshold.

[0378] In some embodiments, the fusion method is centralized fusion if at least one of the following conditions is met:

[0379] The average access duration of at least one sensing device is greater than the second preset duration;

[0380] At least one sensing device contains a sensor that supports velocity / acceleration measurement;

[0381] At least one sensing device has a clock accuracy greater than the third threshold;

[0382] The core network computing power is greater than or equal to the fourth threshold.

[0383] In some embodiments, the control information is determined based on the first request information of the first computing node;

[0384] The first request information is used to request that at least one sensing device be connected to the sensor network, and the first computing node has the ability to perform basic sensing processing and data standardization processing on the raw data collected by at least one sensing device.

[0385] In some embodiments, the first request information of the first computing node includes at least one of the following:

[0386] The computing power of the first computing node;

[0387] The access duration of at least one sensing device;

[0388] The type of sensor present in at least one sensing device;

[0389] The clock accuracy of at least one sensing device;

[0390] The sensing area covered by the first computing node.

[0391] In some embodiments, when the fusion method is distributed fusion, the communication module 71 is further configured to obtain the first sensing data from the second computing node; the second computing node has the ability to process the raw data collected by at least one sensing node to obtain the first sensing data;

[0392] The communication module 71 is also used to send target perception data from the first perception data to each first computing node, wherein the target perception data is the perception data in the first perception data corresponding to the perception area covered by the first computing node; the first computing node has the ability to process the raw data collected by at least one sensing device to obtain the first perception data.

[0393] The communication module 71 is also used to receive third sensing data sent by each first computing node; wherein the third sensing data sent by the first computing node is obtained by the first computing node performing timestamp alignment and correlation fusion on the target sensing data and the second sensing data processed by the first computing node.

[0394] The processing module 72 is also used to spatially fuse the third sensing data of each first computing node to obtain fused sensing data.

[0395] In some embodiments, the third sensing data is obtained by resampling the third sensing data of the first computing node according to a pre-configured sensing refresh rate.

[0396] In some embodiments, the communication module 71 is further configured to receive third sensing data sent by each first computing node based on a pre-configured sensing refresh rate.

[0397] In some embodiments, when the fusion method is centralized fusion, the communication module 71 is used to obtain the first sensing data from the second computing node; the second computing node has the ability to process the raw data collected by at least one sensing node to obtain the first sensing data;

[0398] The communication module 71 is also used to acquire second sensing data from the first computing node; the first computing node has the ability to process raw data collected by at least one sensing device to obtain the first sensing data;

[0399] The processing module 72 is also used to perform timestamp alignment and correlation fusion of the first sensing data and the second sensing data to obtain fused sensing data.

[0400] In some embodiments, the processing module 72 is further configured to resample the fused sensing data based on a pre-configured sensing refresh rate; the communication module 71 is further configured to report the resampled fused sensing data.

[0401] In some embodiments, the pre-configured perceived refresh rate is carried in the control information.

[0402] In some embodiments, association fusion includes at least one of the following:

[0403] The sensing data with the highest reflected signal power among the sensing data corresponding to the same sensing object is taken as the fused data of the same sensing object.

[0404] The sensor data with the highest confidence among the sensor data corresponding to the same sensor object is taken as the fused data of the same sensor object.

[0405] The sensing data corresponding to the same sensing object are weighted and merged based on the reflected signal power to form the fused data of the same sensing object;

[0406] The sensory data corresponding to the same sensory object are weighted and merged based on confidence level to form the fused data of the same sensory object;

[0407] The sensory data corresponding to the same sensory object are combined by weighting the arithmetic average and merging them into the fused data of the same sensory object.

[0408] In some embodiments, if the difference between the perceived data from at least two perceived objects is less than a threshold, the at least two perceived objects are the same perceived object.

[0409] For a more detailed description of the communication module 71 and the processing module 72, as well as a more detailed description of their respective technical features and beneficial effects, please refer to the corresponding method embodiment section above, which will not be repeated here.

[0410] Figure 11 This is a schematic diagram of a communication device provided in an embodiment of the present disclosure. The communication device is applied to a first computing node. The communication device 80 includes a communication module 81 and a processing module 82.

[0411] Communication module 81 is used to receive control information, which is used to instruct the fusion of first sensing data and second sensing data; wherein, the first sensing data comes from at least one sensing node, and the second sensing data is data that meets the requirements of the sensing network, which is obtained by the first computing node through preprocessing of the raw data collected by at least one sensing device.

[0412] Processing module 82 is used to perform perception fusion operations based on control information.

[0413] In some embodiments, the control information includes fusion mode information, and the processing module 82 is specifically used for:

[0414] The fusion method is determined based on the fusion method information in the control information, wherein the fusion method information includes at least one of the following: timestamp alignment method, computing power distribution, and multimodal fusion type;

[0415] Perceptual fusion operations are performed based on the fusion method.

[0416] In some embodiments, when the fusion method is distributed fusion, the communication module 81 is used to receive target perception data from the first perception data obtained from the second computing node and sent by the first node; the target perception data is the perception data in the first perception data corresponding to the perception area covered by the first computing node; the second computing node has the ability to process the raw data collected by at least one sensor node to obtain the first perception data.

[0417] Processing module 82 is used to perform timestamp alignment and correlation fusion on target perception data and second perception data to obtain third perception data;

[0418] The communication module 81 is also used to send third-sensing data to the first node.

[0419] In some embodiments, the third sensing data is obtained by resampling the third sensing data of the first computing node according to a pre-configured sensing refresh rate.

[0420] In some embodiments, the communication module 81 is used to send third sensing data to the first node based on a pre-configured sensing refresh rate.

[0421] In some embodiments, the communication module 81 is used to send second sensing data to the first node.

[0422] For a more detailed description of the communication module 81 and the processing module 82, as well as a more detailed description of their respective technical features and beneficial effects, please refer to the corresponding method embodiment section above, which will not be repeated here.

[0423] It should be noted that, Figure 10 or Figure 11 Modules in a module can also be called units; for example, a communication module can be called a communication unit. Additionally, in... Figure 10 or Figure 11 In the embodiments shown, the names of the modules may not be the same as those shown in the figures. For example, the communication module may also be called the sending module or the receiving module.

[0424] Figure 10 or Figure 11If the various units or modules in the present disclosure are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this disclosure, or the parts that contribute to related technologies, or all or part of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this disclosure. Storage media for storing computer software products include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0425] In implementing the functions of the integrated modules described above using hardware, this disclosure also provides a possible structure for a communication device used to execute the perception fusion method provided in this disclosure. Figure 12 As shown, the communication device 900 includes a communication interface 903, a processor 902, and a bus 904. Optionally, the communication device may also include a memory 901.

[0426] Processor 902 may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with embodiments of this disclosure. Processor 902 may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with embodiments of this disclosure. Processor 902 may also be a combination that implements computing functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0427] The communication interface 903 is used to connect to other devices via a communication network. This communication network can be Ethernet, wireless access network, wireless local area network (WLAN), etc.

[0428] The memory 901 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0429] In one possible implementation, the memory 901 can exist independently of the processor 902. The memory 901 can be connected to the processor 902 via a bus 904 and is used to store instructions or program code. When the processor 902 calls and executes the instructions or program code stored in the memory 901, it can implement the perceptual fusion method provided in the embodiments of this disclosure.

[0430] In another possible implementation, the memory 901 can also be integrated with the processor 902.

[0431] The 904 bus can be an extended industry standard architecture (EISA) bus, etc. The 904 bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 12 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0432] Some embodiments of this disclosure provide a computer-readable storage medium (e.g., a non-transitory computer-readable storage medium) storing computer program instructions that, when executed on a computer, cause the computer to perform a perceptual fusion method as described in any of the above embodiments.

[0433] In one exemplary embodiment, the computer may be the aforementioned communication device, and this disclosure does not limit the specific form of the computer.

[0434] In some examples, the aforementioned computer-readable storage media may include, but are not limited to: magnetic storage devices (e.g., hard disks, floppy disks, or magnetic tapes), optical discs (e.g., compact disks (CDs), digital versatile disks (DVDs), etc.), smart cards, and flash memory devices (e.g., erasable programmable read-only memory (EPROMs), cards, sticks, or key drives, etc.). The various computer-readable storage media described in this disclosure may represent one or more devices and / or other machine-readable storage media for storing information. The term "machine-readable storage media" may include, but is not limited to, wireless channels and various other media capable of storing, containing, and / or carrying instructions and / or data.

[0435] This disclosure provides a computer program product containing instructions that, when run on a computer, cause the computer to execute the perceptual fusion method of any of the above embodiments.

[0436] The above are merely specific embodiments of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any changes or substitutions within the technical scope disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A perceptual fusion method, characterized in that, Applied to the first node, the method includes: Receive control information, the control information being used to instruct the fusion of first sensing data and second sensing data; wherein, the first sensing data originates from at least one sensing node, and the second sensing data is data that meets the requirements of a sensing network, obtained by preprocessing raw data collected by at least one sensing device. The first sensing data and the second sensing data are fused based on the control information to obtain fused sensing data.

2. The method according to claim 1, characterized in that, The control information includes fusion method information. The process of fusing the first sensing data and the second sensing data based on the control information to obtain fused sensing data includes: The fusion method is determined based on the fusion method information in the control information, wherein the fusion method information includes at least one of the following: timestamp alignment method, computing power distribution, and multimodal fusion type; The first sensing data and the second sensing data are fused based on the fusion method to obtain fused sensing data.

3. The method according to claim 2, characterized in that, The fusion method information includes a timestamp alignment method. The fusion method is determined based on the fusion method information in the control information, including: The timestamp alignment method is aligned to the timestamp of the second sensed data, and the fusion method is determined to be distributed fusion. The timestamp alignment method is aligned to the timestamp of the first sensed data, and the fusion method is determined to be centralized fusion.

4. The method according to claim 2, characterized in that, The fusion method information includes computing power distribution. Determining the fusion method based on the fusion method information in the control information includes: In the computing power distribution scenario, the first computing node is responsible for multimodal fusion, and the fusion method is determined to be distributed fusion; the first computing node has the ability to process the raw data collected by at least one sensing device to obtain the second sensing data; In the computing power distribution scenario, the first node is responsible for multimodal fusion, and the fusion method is determined to be centralized fusion.

5. The method according to claim 2, characterized in that, The fusion method is a distributed fusion if at least one of the following conditions is met: The computing power of the first computing node is greater than or equal to the first threshold; the first computing node has the ability to process the raw data collected by at least one sensing device to obtain the second sensing data; The average access time of the at least one sensing device is less than the first preset time. The at least one sensing device contains a sensor that lacks speed sensing capability; Among the at least one sensing device, there is a sensing device whose clock accuracy is less than the second threshold.

6. The method according to claim 2, characterized in that, The fusion method is a centralized fusion if at least one of the following conditions is met: The average access duration of the at least one sensing device is greater than the second preset duration; The at least one sensing device contains a sensor that supports velocity / acceleration measurement; Among the at least one sensing device, there is a sensing device whose clock accuracy is greater than the third threshold; The core network computing power is greater than or equal to the fourth threshold.

7. The method according to claim 1, characterized in that, The control information is determined based on the first request information of the first computing node; The first request information is used to request the connection of the at least one sensing device to the sensor network, and the first computing node has the ability to perform basic sensing processing and data standardization processing on the raw data collected by the at least one sensing device.

8. The method according to claim 7, characterized in that, The first request information of the first computing node includes at least one of the following: The computing power of the first computing node; The access duration of the at least one sensing device; The type of sensor present in the at least one sensing device; The clock accuracy of the at least one sensing device; The sensing area covered by the first computing node.

9. The method according to claim 2, characterized in that, When the fusion method is distributed fusion, the step of fusing the first sensing data and the second sensing data based on the fusion method to obtain fused sensing data includes: The first sensing data is obtained from the second computing node; the second computing node has the ability to process the raw data collected by at least one sensing node to obtain the first sensing data; For each first computing node, target sensing data from the first sensing data is sent to the first computing node. The target sensing data is the sensing data in the first sensing data that corresponds to the sensing area covered by the first computing node. The first computing node has the ability to process the raw data collected by at least one sensing device to obtain the first sensing data. Receive third sensing data sent by each first computing node; wherein, the third sensing data sent by the first computing node is obtained by the first computing node performing timestamp alignment and correlation fusion on the target sensing data and the second sensing data processed by the first computing node; Spatial fusion is performed on the third sensing data of each of the first computing nodes to obtain the fused sensing data.

10. The method according to claim 9, characterized in that, The third sensing data is obtained by resampling the third sensing data of the first computing node according to the pre-configured sensing refresh rate.

11. The method according to claim 9, characterized in that, The receiving of third-sensing data sent by each first computing node includes: The third sensing data is received from each first computing node based on a pre-configured sensing refresh rate.

12. The method according to claim 2, characterized in that, When the fusion method is centralized fusion, the step of fusing the first sensing data and the second sensing data based on the fusion method to obtain fused sensing data includes: The first sensing data is obtained from the second computing node; the second computing node has the ability to process the raw data collected by at least one sensing node to obtain the first sensing data; The second sensing data is obtained from the first computing node; the first computing node has the ability to process the raw data collected by at least one sensing device to obtain the first sensing data; The first sensing data and the second sensing data are time-stamp aligned and correlated to obtain the fused sensing data.

13. The method according to claim 12, characterized in that, The method further includes: The fused sensing data is resampled based on a pre-configured sensing refresh rate; Report the fused sensing data after resampling.

14. The method according to any one of claims 10, 11, or 12, characterized in that, The pre-configured perceived refresh rate is carried in the control information.

15. The method according to claim 9 or 12, characterized in that, The association fusion includes at least one of the following: The sensing data with the highest reflected signal power among the sensing data corresponding to the same sensing object is taken as the fused data of the same sensing object. The perception data with the highest confidence among the perception data corresponding to the same perception object is taken as the fused data of the same perception object. The sensing data corresponding to the same sensing object are weighted and merged based on the reflected signal power to obtain the fused data of the same sensing object; The sensory data corresponding to the same sensory object are weighted and merged based on confidence level to obtain the fused data of the same sensory object; The sensory data corresponding to the same sensory object are combined by weighting the arithmetic mean and then used as the fused data of the same sensory object.

16. The method according to claim 15, characterized in that, If the difference between the perceived data from at least two perceived objects is less than a threshold, the at least two perceived objects are considered to be the same perceived object.

17. A perceptual fusion method, characterized in that, Applied to the first computing node, the method includes: The system receives control information, which is used to instruct the fusion of first sensing data and second sensing data; wherein the first sensing data originates from at least one sensing node, and the second sensing data is data that meets the requirements of the sensing network, obtained by the first computing node through preprocessing of the raw data collected by at least one sensing device. Perform a perception fusion operation based on the control information.

18. The method according to claim 17, characterized in that, The control information includes fusion method information, and the step of performing a perception fusion operation based on the control information includes: The fusion method is determined based on the fusion method information in the control information, wherein the fusion method information includes at least one of the following: timestamp alignment method, computing power distribution, and multimodal fusion type; Perform a perception fusion operation based on the aforementioned fusion method.

19. The method according to claim 18, characterized in that, When the fusion method is distributed fusion, the step of performing the perception fusion operation based on the fusion method includes: The first node receives target sensing data from the first sensing data obtained from the second computing node; the target sensing data is the sensing data in the first sensing data that corresponds to the sensing area covered by the first computing node; the second computing node has the ability to process the raw data collected by at least one sensor node to obtain the first sensing data. The target perception data and the second perception data are time-stamp aligned and fused to obtain the third perception data; The third sensing data is sent to the first node.

20. The method according to claim 19, characterized in that, The third sensing data is obtained by resampling the third sensing data of the first computing node according to the pre-configured sensing refresh rate.

21. The method according to claim 19, characterized in that, Sending the third sensing data to the first node includes: The third sensing data is sent to the first node based on the pre-configured sensing refresh rate.

22. The method according to claim 18, characterized in that, When the fusion method is centralized fusion, the step of performing the perception fusion operation based on the fusion method includes: The second sensing data is sent to the first node.

23. A communication device, characterized in that, include: Memory and processor; Memory and processor are coupled; The memory is used to store instructions that can be executed by the processor; When the processor executes the instructions, it performs the method as described in any one of claims 1 to 22.

24. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed on a communication device, cause the communication device to perform the method as described in any one of claims 1 to 22.

25. A computer program product, characterized in that, When the computer program product is executed, it implements the method as described in any one of claims 1 to 22.