Communication method, device, system and storage medium

Abstract

The present application belongs to the field of communications. Disclosed are a communication method, a device, a system and a storage medium. The system comprises a processing device and a plurality of sensing devices; each sensing device comprises a first optical interface; the processing device comprises a second optical interface; the first optical interfaces of the sensing devices are connected to the second optical interface of the processing device by means of a point-to-multipoint optical connection; each sensing device is used for acquiring first sensing data; each sensing device is further used for sending the first sensing data to the processing device by means of the optical connection; and the processing device performs target processing on the first sensing data to obtain second sensing data. The present application improves the quality of sensing data while reducing the cost of the sensing devices.

Description

Communication methods, devices, systems and storage media

[0001] This application claims priority to Chinese Patent Application No. 202411419847.3, filed on October 11, 2024, entitled "Communication Method, Apparatus, System and Storage Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of communications, and in particular to a communication method, device, system and storage medium. Background Technology

[0003] Sensing devices are widely used in people's daily lives, including clothing, food, housing, and transportation. A sensing device can sense its environment to obtain initial sensing data, then perform enhancement, noise reduction, detection, and / or recognition processing on this initial sensing data to obtain second sensing data, which is then transmitted.

[0004] For example, the sensing device can be a camera, which includes an image sensor, a processing chip, and a network interface. The image sensor exposes the environment in which the camera is located to obtain first image data. The processing chip performs enhancement, de-manipulation, detection, and / or recognition processing on the first image data to obtain second image data. The network interface can send the second image data.

[0005] As people's demands for the functionality of sensing devices gradually increase, they also require their prices to decrease. To reduce the price of sensing devices, fewer computing resources are deployed within them. This can result in insufficient computing power to perform enhancement, noise reduction, detection, and / or recognition processing on the initial sensing data, leading to lower-quality secondary sensing data.

[0006] For example, taking a camera as a sensing device, to reduce the price of the camera, a lower-performance processing chip is deployed in it. In scenes with high pedestrian density, the first image data obtained by the image sensor includes a large number of target images, resulting in a large amount of data to process. However, due to the limited computing power provided by the processing chip, it lacks sufficient computing power to process the first image data, resulting in lower-quality second image data. Summary of the Invention

[0007] This application provides a communication method, device, system, and storage medium to improve the quality of sensed data while reducing the cost of sensing devices. The technical solution is as follows:

[0008] In a first aspect, this application provides a communication and sensing integrated system, the system including a processing device and a plurality of sensing devices. For each sensing device, the sensing device includes a first optical interface and the processing device includes a second optical interface. The first optical interface of the sensing device is connected to the second optical interface of the processing device through a point-to-multipoint optical connection.

[0009] Sensing devices are used to acquire initial sensing data;

[0010] The sensing device is also used to send first sensing data to the processing device via an optical connection;

[0011] The processing device is used to perform target processing on the first sensing data to obtain the second sensing data.

[0012] Because the sensing device sends the first sensing data to the processing device via an optical connection, and the processing device performs target processing on the first sensing data to obtain the second sensing data, the target processing is transferred from the sensing device to the processing device. The processing device has sufficient computing resources to perform target processing on the first sensing data, resulting in high-quality second sensing data. The sensing device, on the other hand, does not perform target processing. Therefore, there is no need to deploy computing resources for target processing on the sensing device. This not only reduces the cost of the sensing device but also improves the quality of the second sensing data, thus achieving both cost reduction and improved sensing data quality.

[0013] In one possible implementation, the first sensing data includes background data and foreground data. The background data is the same as the third sensing data and the first sensing data, and the foreground data is the different from the third sensing data and the first sensing data. The third sensing data is the previous frame data acquired by the sensing device.

[0014] In another possible implementation, the sensing device is used to compress foreground data based on a first compression ratio to obtain first compressed data, and compress background data based on a second compression ratio to obtain second compressed data, wherein the first compression ratio is less than the second compression ratio; and to send the first compressed data and the second compressed data to the processing device via an optical connection.

[0015] The processing device is also used to decompress the first compressed data to obtain foreground data, and to decompress the second compressed data to obtain background data; based on the foreground data and the background data, the first sensing data is obtained.

[0016] Because the first compression ratio is lower than the second compression ratio, and the sensing device compresses foreground data based on the first compression ratio to obtain first compressed data, and compresses background data based on the second compression ratio to obtain second compressed data, the compression depth of the foreground data is less than the compression depth of the background data. In other words, shallow compression is applied to the foreground data to retain as much information as possible, while deep compression is applied to the background data to reduce the amount of data in the second compressed data. This reduces the bandwidth consumption of the optical connection, allowing more sensing devices in the integrated communication and sensing system to connect to the processing device, meaning more sensing devices can send sensing data to the processing device.

[0017] In another possible implementation, the processing device includes background data. A sensing device is used to obtain first compressed data by compressing the foreground data based on a first compression ratio; and to transmit the first compressed data to the processing device via an optical connection.

[0018] The processing device is also used to decompress the first compressed data to obtain foreground data; and to obtain the first perceived data based on the foreground data and background data.

[0019] Since the processing device includes background data, the sensing device compresses the foreground data based on a first compression ratio to obtain first compressed data, and sends the first compressed data to the processing device via an optical connection. The processing device decompresses the first compressed data to obtain the foreground data, and obtains the first sensing data based on the foreground data and background data. Therefore, the sensing device does not need to send background data, reducing the bandwidth resource consumption of the optical connection. This allows more sensing devices in the integrated communication and sensing system to connect to the processing device, meaning more sensing devices can send sensing data to the processing device.

[0020] In another possible implementation, the first compression ratio is determined based on the actual bandwidth available to the sensing device. This allows the compression ratio to dynamically change with the actual bandwidth available to the sensing device, ensuring that the bandwidth actually available to the sensing device meets the bandwidth required to send the compressed data.

[0021] In another possible implementation, the processing device includes fourth sensing data, which is the data previously received by the processing device. There is at least one frame of sensing data between the first sensing data and the fourth sensing data. The processing device is also used to acquire at least one frame of fifth sensing data located between the fourth sensing data and the first sensing data based on the change pattern of the foreground data in the fourth sensing data, the first sensing data, and the fourth sensing data; and to perform target processing on the at least one frame of fifth sensing data.

[0022] This eliminates the need for sensing devices to send every frame of sensing data, reducing the network resource consumption of optical connections and allowing more sensing devices in the integrated communication and sensing system to connect to the processing equipment.

[0023] In another possible implementation, a processing device is used to allocate computing resources based on the amount of foreground data; and to perform target processing on the first sensing data based on the allocated computing resources to obtain the second sensing data.

[0024] This way, computing resources are allocated only according to the required amount, which avoids wasting computing resources and ensures the quality of target processing on the first perception data, thereby improving the quality of the second perception data obtained from the target processing.

[0025] In another possible implementation, the processing device is also used to allocate bandwidth to each sensing device based on the bandwidth required by each sensing device and the total bandwidth of the second optical interface. This allows the processing device to uniformly allocate bandwidth to each sensing device, ensuring that more sensing devices in the integrated communication and sensing system can connect to the processing device, i.e., enabling more sensing devices to send sensing data to the processing device.

[0026] In another possible implementation, target processing includes one or more of the following: enhancement processing, denoising processing, target detection processing, target recognition processing, perceptual data fusion, or perceptual data stitching.

[0027] In another possible implementation, the sensing device is a camera device, and the first sensing data is image data; or, the sensing device is a radar device, and the first sensing data is point cloud data; or, the sensing device is an audio acquisition device, and the first sensing data is audio data.

[0028] In another possible implementation, the communication-sensing integrated system is either a passive optical network (PON) or a fiber-to-the-room (FTTR) network.

[0029] Secondly, this application provides a communication method applied to a communication-sensing integrated system. The system includes a processing device and multiple sensing devices. For each sensing device, the sensing device includes a first optical interface, and the processing device includes a second optical interface. The first optical interface of the sensing device is connected to the second optical interface of the processing device via a point-to-multipoint optical connection. In the method, the sensing device acquires first sensing data; the sensing device sends the first sensing data to the processing device via the optical connection. The processing device performs target processing on the first sensing data to obtain second sensing data.

[0030] Because the sensing device sends the first sensing data to the processing device via an optical connection, and the processing device performs target processing on the first sensing data to obtain the second sensing data, the target processing is transferred from the sensing device to the processing device. The processing device has sufficient computing resources to perform target processing on the first sensing data, resulting in high-quality second sensing data. The sensing device, on the other hand, does not perform target processing. Therefore, there is no need to deploy computing resources for target processing on the sensing device. This not only reduces the cost of the sensing device but also improves the quality of the second sensing data, thus achieving both cost reduction and improved sensing data quality.

[0031] In one possible implementation, the first sensing data includes background data and foreground data. The background data is the same as the third sensing data and the first sensing data, and the foreground data is the different data between the third sensing data and the first sensing data. The third sensing data is the previous frame data acquired by the sensing device.

[0032] In another possible implementation, the sensing device compresses foreground data based on a first compression ratio to obtain first compressed data, and compresses background data based on a second compression ratio to obtain second compressed data, where the first compression ratio is less than the second compression ratio. The sensing device sends the first and second compressed data to a processing device via an optical connection. The processing device decompresses the first compressed data to obtain foreground data, and decompresses the second compressed data to obtain background data. Based on the foreground and background data, the processing device obtains first sensing data.

[0033] Because the first compression ratio is lower than the second compression ratio, and the sensing device compresses foreground data based on the first compression ratio to obtain first compressed data, and compresses background data based on the second compression ratio to obtain second compressed data, the compression depth of the foreground data is less than the compression depth of the background data. In other words, shallow compression is applied to the foreground data to retain as much information as possible, while deep compression is applied to the background data to reduce the amount of data in the second compressed data. This reduces the bandwidth consumption of the optical connection, allowing more sensing devices in the integrated communication and sensing system to connect to the processing device, meaning more sensing devices can send sensing data to the processing device.

[0034] In another possible implementation, the processing device includes background data. The sensing device compresses the foreground data based on a first compression ratio to obtain first compressed data. The sensing device sends the first compressed data to the processing device via an optical connection. The processing device decompresses the first compressed data to obtain the foreground data. Based on the foreground data and the background data, the processing device obtains first perceived data.

[0035] Since the processing device includes background data, the sensing device compresses the foreground data based on a first compression ratio to obtain first compressed data, and sends the first compressed data to the processing device via an optical connection. The processing device decompresses the first compressed data to obtain the foreground data, and obtains the first sensing data based on the foreground data and background data. Therefore, the sensing device does not need to send background data, reducing the bandwidth resource consumption of the optical connection. This allows more sensing devices in the integrated communication and sensing system to connect to the processing device, meaning more sensing devices can send sensing data to the processing device.

[0036] In another possible implementation, the first compression ratio is determined based on the actual bandwidth available to the sensing device. This allows the compression ratio to dynamically change with the actual bandwidth available to the sensing device, ensuring that the bandwidth actually available to the sensing device meets the bandwidth required to send the compressed data.

[0037] In another possible implementation, the processing device includes fourth sensing data, which is the data previously received by the processing device, with at least one frame of sensing data between the first and fourth sensing data. Based on the changing patterns of foreground data in the fourth sensing data, the first sensing data, and the fourth sensing data, the processing device acquires at least one frame of fifth sensing data located between the fourth and first sensing data. The processing device performs target processing on this at least one frame of fifth sensing data. This eliminates the need for the sensing devices to send every frame of sensing data they perceive, reducing the network resource consumption of optical connections and allowing more sensing devices in a communication-sensing integrated system to connect to the processing device.

[0038] In another possible implementation, the processing device allocates computing resources based on the amount of foreground data. The processing device then performs target processing on the first sensed data based on the allocated computing resources to obtain the second sensed data. This approach allocates computing resources only as needed, avoiding waste while ensuring the quality of target processing on the first sensed data and improving the quality of the resulting second sensed data.

[0039] In another possible implementation, the processing device allocates bandwidth to each sensing device based on the bandwidth required by each sensing device and the total bandwidth of the second optical interface. This allows the processing device to uniformly allocate bandwidth to each sensing device, ensuring that more sensing devices in the integrated communication and sensing system can connect to the processing device, i.e., enabling more sensing devices to send sensing data to the processing device.

[0040] In another possible implementation, target processing includes one or more of the following: enhancement processing, denoising processing, target detection processing, target recognition processing, perceptual data fusion, or perceptual data stitching.

[0041] In another possible implementation, the sensing device is a camera device, and the first sensing data is image data; or, the sensing device is a radar device, and the first sensing data is point cloud data; or, the sensing device is an audio acquisition device, and the first sensing data is audio data.

[0042] In another possible implementation, the sensing device is a camera device, which includes a lens and an image sensor.

[0043] In another possible implementation, the communication-sensing integrated system is either a passive optical network (PON) or a fiber-to-the-room (FTTR) network.

[0044] Thirdly, this application provides a sensing device, which includes a sensing module, a conversion module, and an optical interface;

[0045] The sensing module is used to acquire sensing data;

[0046] The conversion module is used to convert the sensed data into a target signal in the format defined by the target communication protocol, and to convert the target signal into an optical signal. The target communication protocol is the communication protocol adopted by the sensing device.

[0047] An optical interface is used to transmit optical signals.

[0048] After the sensing module acquires the sensing data, the conversion module converts the sensing data into an optical signal, which is then transmitted via the optical interface. This eliminates the need for target processing within the sensing device, reducing its cost. The conversion module converts the sensing data into a target signal, which in turn converts it into an optical signal. The target signal is a signal defined by the communication protocol used by the sensing device, enabling it to access the optical network. The optical signal has a larger bandwidth and higher transmission efficiency, thus increasing the sensing device's transmission bandwidth and supporting high-definition frame data transmission.

[0049] In one possible implementation, the target communication protocol includes one or more of the following: Passive Optical Network (PON) protocol, Fiber to the Room (FTTR) protocol, or Ethernet protocol.

[0050] In another possible implementation, the sensing device also includes a compression module, which is used to perform shallow compression on the sensing data to obtain compressed data;

[0051] The conversion module converts compressed data into the target signal and the target signal into an optical signal. This shallow compression of the sensed data preserves as much information as possible while minimizing the bandwidth consumption of the optical network.

[0052] In another possible implementation, the sensing device also includes a wireless communication module for generating a wireless local area network (WLAN). This allows the sensing device to act as an access point, enabling other devices to connect and expanding the coverage of the integrated communication and sensing system.

[0053] In another possible implementation, the sensing device also includes a power module for providing power to the sensing module and the conversion module.

[0054] In another possible implementation, the optical interface is a photoelectric composite interface, which also provides power to the sensing module and conversion module. This eliminates the need for a separate power supply module, reducing the size of the sensing device. Furthermore, in scenarios where power supply is inconvenient, power can be remotely supplied to the modules in the sensing device via the optical interface, improving power supply flexibility and expanding application scenarios.

[0055] In another possible implementation, the sensing device is a camera, and the sensing data is image data; or...

[0056] The sensing device is a radar device, and the sensing data is point cloud data; or...

[0057] The sensing device is an audio acquisition device, and the sensing data is audio data.

[0058] Fourthly, this application provides a processing device, which is part of a communication and sensing integrated system. The system also includes multiple sensing devices, each of which includes a first optical interface. The processing device includes a second optical interface, a conversion module, and a processing unit. The second optical interface is connected to the first optical interface of each sensing device through a point-to-multipoint optical connection.

[0059] The second optical interface is used to receive at least one optical signal via an optical connection. The at least one optical signal is obtained by at least one sensing device acquiring at least one first sensing data and converting the at least one first sensing data. The multiple sensing devices include at least one sensing device.

[0060] A conversion module for converting at least one optical signal into at least one first sensing data;

[0061] The processing unit is used to perform target processing on at least one first sensing data to obtain at least one second sensing data.

[0062] Because the processing device receives the optical signals sent by each sensing device, converts the optical signals into first sensing data, and performs target processing on the first sensing data to obtain second sensing data, the target processing is migrated from the sensing devices to the processing device. The processing device has sufficient computing resources to perform target processing on the first sensing data, resulting in high-quality second sensing data. The sensing devices, on the other hand, do not perform target processing. Therefore, there is no need to deploy computing resources for target processing on the sensing devices. This not only reduces the cost of the sensing devices but also improves the quality of the second sensing data, thus achieving both reduced cost and improved sensing data quality.

[0063] In one possible implementation, the processing device also includes: a scheduling unit and a computing resource pool;

[0064] For each piece of first-sensing data, the scheduling unit allocates computing resources from the computing resource pool based on the amount of data in the first-sensing data.

[0065] The processing unit is used to perform target processing on the first sensing data based on the allocated computing resources.

[0066] This way, computing resources are allocated only according to the required amount, which avoids wasting computing resources and ensures the quality of target processing on the first perception data, thereby improving the quality of the second perception data obtained from the target processing.

[0067] In another possible implementation, the processing unit is also used to allocate bandwidth to each sensing device based on the bandwidth required by each sensing device and the total bandwidth of the second optical interface. This allows the processing device to uniformly allocate bandwidth to each sensing device, ensuring that more sensing devices in the integrated communication and sensing system can connect to the processing device, i.e., enabling more sensing devices to send sensing data to the processing device.

[0068] In another possible implementation, at least one optical signal is obtained by converting at least one compressed data, which is obtained by shallowly compressing at least one first sensing data;

[0069] A conversion module for converting at least one optical signal into at least one compressed data;

[0070] The processing unit is also configured to decompress at least one compressed data to obtain at least one first sensing data.

[0071] Because the processing unit can decompress, the sensing device can perform shallow compression on the sensing data to retain as much information as possible and reduce the bandwidth consumption of the optical network.

[0072] In another possible implementation, target processing includes one or more of the following: enhancement processing, denoising processing, target detection processing, target recognition processing, perceptual data fusion, or perceptual data stitching.

[0073] In another possible implementation, the second optical interface is a photoelectric composite interface, which also serves to supply power to multiple sensing devices. This eliminates the need for separate power modules in the sensing devices, reducing their size. Furthermore, in scenarios where power supply is inconvenient, power can be supplied to the sensing devices remotely via the second optical interface, improving power supply flexibility and expanding application scenarios.

[0074] In another possible implementation, the system is a passive optical network (PON) or a fiber-to-the-room (FTTR) network.

[0075] Fifthly, this application provides a computing device cluster, the computing device cluster including at least one computing device, each computing device including a processor and a memory;

[0076] The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method of the first aspect or any possible implementation thereof.

[0077] In a sixth aspect, this application provides a computer program product containing instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method of the first aspect or any possible implementation thereof.

[0078] In a seventh aspect, this application provides a computer-readable storage medium including computer program instructions that, when executed by a cluster of computing devices, perform the method of the first aspect or any possible implementation thereof.

[0079] Eighthly, this application provides a chip including a memory and a processor, wherein the memory is used to store computer instructions, and the processor is used to call and execute the computer instructions from the memory to perform the method in the first aspect or any possible implementation thereof. Attached Figure Description

[0080] Figure 1 is a schematic diagram of the structure of a communication and sensing integrated system provided in an embodiment of this application;

[0081] Figure 2 is a schematic diagram of another integrated communication and sensing system provided in an embodiment of this application;

[0082] Figure 3 is a schematic diagram of another integrated communication and sensing system provided in an embodiment of this application;

[0083] Figure 4 is a schematic diagram of the structure of a sensing device provided in an embodiment of this application;

[0084] Figure 5 is a schematic diagram of another sensing device provided in an embodiment of this application;

[0085] Figure 6 is a schematic diagram of the structure of a processing device provided in an embodiment of this application;

[0086] Figure 7 is a schematic diagram of another processing device provided in an embodiment of this application;

[0087] Figure 8 is a flowchart of a communication method provided in an embodiment of this application;

[0088] Figure 9 is a schematic diagram of another sensing device provided in an embodiment of this application;

[0089] Figure 10 is a schematic diagram of another processing device provided in an embodiment of this application;

[0090] Figure 11 is a flowchart of another communication method provided in an embodiment of this application;

[0091] Figure 12 is a schematic diagram of another integrated communication and sensing system provided in an embodiment of this application;

[0092] Figure 13 is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0093] Figure 14 is a schematic diagram of a cluster structure integrating communication and sensing provided in an embodiment of this application;

[0094] Figure 15 is a schematic diagram of another integrated communication and sensing cluster structure provided in an embodiment of this application. Detailed Implementation

[0095] The sensing device is used to sense its surrounding environment to obtain first sensing data, and then to process the first sensing data to obtain second sensing data.

[0096] Optionally, target processing includes one or more of the following: enhancement processing, noise reduction processing, target detection processing, target recognition processing, perceptual data fusion, or perceptual data stitching.

[0097] For example, when the sensing device is a camera, the camera includes a lens, an image sensor, a processing chip, and an Ethernet interface. The image sensor is used to expose the environment around the camera to obtain first image data. The processing chip performs enhancement and noise reduction processing on the first image data to obtain second image data. The second image data can be sent through the Ethernet interface. The user uses the device to receive and display the second image data for viewing.

[0098] The sensing device performs target processing on the first sensed data, which requires the sensing device to include computing resources for target processing (such as the processing chip in the aforementioned camera device), increasing the cost of the sensing device.

[0099] The environment in which the sensing devices operate changes in real time, causing the amount of computing resources required to process the initial sensing data to fluctuate. However, the amount of computing resources within the sensing devices themselves remains constant. This means that sometimes the sensing devices have fewer computing resources than are needed to process the initial sensing data, resulting in lower-quality second-sensing data. Conversely, sometimes the sensing devices have more computing resources than are needed to process the initial sensing data, leading to significant wasted computing resources during the initial processing.

[0100] For example, taking a camera as a sensing device, the processing chip in the camera is a computing resource. After the first image data is obtained through exposure, the processing chip needs to be used to enhance and denoise the first image data, which increases the cost of the camera.

[0101] In camera equipment, the computing power of the processing chip is fixed. When there are many people around the camera, the first image data obtained by the camera includes a large number of targets that require enhancement and noise reduction. This necessitates significant computing resources for these processes. However, the processing chip may lack sufficient computing power, potentially leading to target loss and missed captures when processing a large number of targets. Alternatively,

[0102] When there are few people around the camera, the first image data obtained by the camera includes a small number of targets that need to be enhanced and denoised. At this time, only a small amount of computing power of the processing chip is needed to enhance and denoise the first image data, resulting in a waste of a large amount of computing power of the processing chip.

[0103] To reduce the cost of sensing devices and avoid situations where sensing devices sometimes have insufficient computing resources and sometimes waste their computing resources, referring to Figure 1, this application provides an integrated communication and sensing system 100. The system 100 includes a processing device 101 and multiple sensing devices 102. For each sensing device 102, the sensing device 102 includes a first optical interface, and the processing device 101 includes a second optical interface. The first optical interface of the sensing device 102 is connected to the second optical interface of the processing device 101 via a point-to-multipoint (P2MP) optical connection. The processing device 101 includes a computing resource pool.

[0104] In the system 100, after each sensing device 102 obtains first sensing data, it sends the first sensing data to the processing device 101 via an optical connection. That is, each sensing device 102 does not perform target processing on the first sensing data, so each sensing device 102 may not include computing resources for target processing.

[0105] For each sensing device 102, the processing device 101 receives first sensing data sent by the sensing device 102, and based on the first sensing data, obtains the amount of computing power resources required to perform target processing on the first sensing data. Based on the amount of computing power resources, it allocates computing power resources from the computing power resource pool, and uses the allocated computing power resources to perform target processing on the first sensing data to obtain second sensing data.

[0106] When the amount of computing resources required for target processing of the first perception data is large, allocate more computing resources to ensure that there are enough computing resources to process the first perception data and obtain high-quality second perception data.

[0107] When the amount of computing resources required for target processing of the first sensing data is small, less computing resources are allocated. This allows the excess computing resources to be distributed to other sensing devices that require more computing resources, thus avoiding waste of computing resources.

[0108] Since the sensing device 102 does not include computing resources for target processing, the cost of the sensing device 102 can be saved. Because computing resources are dynamically allocated based on the amount of computing resources required for target processing of the first sensing data from the sensing device 102, insufficient or wasted computing resources are avoided.

[0109] In some embodiments, the P2MP optical connection features high bandwidth and low latency.

[0110] In some embodiments, each sensing device 102 and processing device 101 are clock-synchronized via the optical connection. Because the optical connection features high bandwidth and low latency, precise clock synchronization can be achieved between each sensing device 102 and processing device 101 via the optical connection.

[0111] In some embodiments, referring to FIG2, the system 100 may include only one sensing device 102. The system 100 is a point-to-point (P2P) network. The system 100 includes a processing device 101 and a sensing device 102. The first optical interface of the sensing device 102 and the second optical interface of the processing device 101 are connected by optical fiber.

[0112] In some embodiments, the communication protocol used between the sensing device 102 and the processing device 101 in the system 100 may include one or more of the following: passive optical network (PON) protocol, fiber to the room (FTTR) protocol, or Ethernet protocol.

[0113] In some embodiments, referring to FIG3, the system 100 further includes a beam splitter 103, which is connected to a second optical interface of the processing device 101 via an optical fiber, and the beam splitter 103 is also connected to a first optical interface of a plurality of sensing devices 102 via multiple optical fibers.

[0114] Optionally, the optical fiber can be a photoelectric composite optical fiber, and both the second optical interface of the processing device 101 and the first optical interface of the sensing device 102 are photoelectric composite interfaces. In this way, the processing device 101 can both communicate with the sensing device 102 using optical fiber and supply power to the sensing device 102 through optical fiber. Alternatively, the optical fiber can be a pure optical fiber, and the processing device 101 does not need to use optical fiber to supply power to the sensing device 102.

[0115] In some embodiments, the plurality of sensing devices 102 may include one or more of the following devices: camera devices, radar devices, or audio acquisition devices, etc.

[0116] The sensing device 102 is a camera device, and the first sensing data and the second sensing data are the first image data and the second image data, respectively.

[0117] The sensing device 102 is a radar device, and the first sensing data and the second sensing data are the first point cloud data and the second point cloud data, respectively.

[0118] The sensing device 102 is an audio acquisition device, and the first sensing data and the second sensing data are the first audio data and the second audio data, respectively. Optionally, the audio acquisition device may include a microphone, etc.

[0119] The plurality of sensing devices 102 may also include other devices, such as infrared sensing devices and / or fiber optic sensing devices, which will not be listed here.

[0120] In some embodiments, the processing device 101 can be a gateway, server, or cloud service platform, etc. Optionally, the system 100 can be an FTTR system, the processing device 101 can be the main gateway in the FTTR system, and multiple sensing devices 102 can be distributed in different rooms of the FTTR system. Alternatively, the system 100 can be a PON system.

[0121] Referring to Figure 4, this application embodiment provides a sensing device 400, which may be a sensing device in the communication and sensing integrated system 100 shown in Figures 1, 2, or 3. The sensing device 400 includes a sensing module 401, a conversion module 402, and an optical interface 403;

[0122] Sensing module 401 is used to acquire sensing data;

[0123] The conversion module 402 is used to convert the sensed data into a target signal in the format defined by the target communication protocol, and to convert the target signal into an optical signal. The target communication protocol is the communication protocol adopted by the sensing device 400.

[0124] Optical interface 403 is used to transmit optical signals.

[0125] In some embodiments, the target communication protocol includes one or more of the following: PON protocol, FTTR protocol, or Ethernet protocol, etc.

[0126] The sensing module 401 is used to obtain sensing data by sensing the environment around the sensing device 400.

[0127] In some embodiments, the sensing module 401 includes a sensing sensor, and the sensing module 401 is used to sense the environment around the sensing device 400 through the sensing sensor to obtain sensing data.

[0128] Optionally, the sensing device 400 is a camera device, and the sensing module 401 may include a lens. The sensing data is image data.

[0129] Optionally, the sensing device 400 is a radar device. The sensing sensor includes a transmitter and a receiver. The transmitter is responsible for modulating and transmitting the signal (such as a millimeter-wave signal), and the receiver is responsible for receiving and demodulating the echo signal. The sensing module 401 is used to detect the surrounding environment of the sensing device 400 through the transmitter and receiver to obtain sensing data, which is point cloud data. Optionally, the radar device can be a millimeter-wave radar or a lidar, etc.

[0130] Optionally, the sensing device 400 is an audio acquisition device, and the sensing sensor includes a microphone and / or an audio receiver, etc. The sensing module 401 is used to collect sound from the environment surrounding the sensing device 400 through the microphone and / or audio receiver to obtain sensing data, which is audio data. Optionally, the audio acquisition device may include a microphone, etc.

[0131] In some embodiments, referring to FIG5, the sensing device 400 further includes a compression module 404.

[0132] Compression module 404 is used to perform shallow compression on the sensed data to obtain compressed data;

[0133] The conversion module 402 is used to convert compressed data into a target signal and the target signal into an optical signal.

[0134] Shallow compression uses a compression ratio less than the compression ratio threshold. The processing device in the aforementioned integrated communication and sensing system uses deep compression to compress the sensing data. The compression ratio used in deep compression is greater than the compression ratio threshold. Therefore, the compression ratio used by the processing device for deep compression is greater than the compression ratio used by the compression module 404 for shallow compression.

[0135] The sensing device 400 may include a small amount of computing resources, enabling the compression module 404 to perform shallow compression on the sensing data. This reduces the amount of data in the optical signal and the bandwidth resources required to transmit the optical signal.

[0136] In some embodiments, referring to FIG5, the sensing device 400 further includes a wireless communication module 405 for generating a wireless local area network.

[0137] In this way, the terminal device can access the wireless local area network and communicate with the sensing device 400. At this time, the sensing device 400 can act as an access point (AP).

[0138] In some embodiments, referring to FIG5, the sensing device 400 further includes a power module 406, which provides power to the sensing module 401 and the conversion module 402.

[0139] Optionally, the power module 406 includes a battery that can provide power to the various modules in the sensing device 400, such as the sensing module 401, the conversion module 402, the compression module 404, and / or the wireless communication module 405. Alternatively, the battery module 406 can be connected to mains power to receive power from the mains and provide power to the sensing module 401, the conversion module 402, the compression module 404, and / or the wireless communication module 405.

[0140] In some embodiments, the optical interface 403 is a photoelectric composite interface, and the optical interface 403 is also used to provide power to the sensing module 401 and the conversion module 402.

[0141] The optical interface 403 can be connected to a processing device in an integrated optical fiber and communication sensing system. The processing device can provide power to the optical interface 403 of the sensing device 400. The optical interface 403 is used to receive power and provide power to the sensing module 401, the conversion module 402, the compression module 404, and / or the wireless communication module 405.

[0142] In this embodiment, the sensing module acquires sensing data; the conversion module converts the sensing data into a target signal in the target communication protocol definition format, and the target signal is converted into an optical signal; the optical interface sends the optical signal. In this way, the sensing device does not need to perform target processing on the sensing data, and thus there is no need to deploy computing resources for target processing on the sensing device, thereby reducing the cost of the sensing device.

[0143] Referring to Figure 6, this application embodiment provides a processing device 600, which is a device included in the communication and sensing integrated system 100 shown in Figures 1, 2 or 3 above. The system 100 also includes multiple sensing devices, each sensing device including a first optical interface. The processing device 600 includes a second optical interface 601, a conversion module 602 and a processing unit 603. The second optical interface 601 is connected to the first optical interface of each sensing device through a point-to-multipoint optical connection.

[0144] The second optical interface 601 is used to receive at least one optical signal through the optical connection. The at least one optical signal is obtained by at least one sensing device acquiring at least one first sensing data and converting the at least one first sensing data. The plurality of sensing devices includes the at least one sensing device.

[0145] The conversion module 602 is used to convert the at least one optical signal into at least one first sensing data;

[0146] The processing unit 603 is used to perform target processing on the at least one first sensing data to obtain at least one second sensing data.

[0147] In some embodiments, referring to FIG7, the processing device 600 further includes: a scheduling unit 604 and a computing resource pool 605;

[0148] For each piece of first-sensing data, the scheduling unit 604 allocates computing resources from the computing resource pool 605 based on the amount of data in the first-sensing data.

[0149] The processing unit 603 is used to perform target processing on the first sensing data based on the allocated computing resources.

[0150] The scheduling unit 604 determines the amount of computing resources required for target processing of the first sensing data based on the first sensing data. Based on this amount of computing resources, it allocates computing resources from the computing resource pool 605. The processing unit 603 then uses the allocated computing resources to perform target processing on the first sensing data to obtain second sensing data. This approach ensures that when the amount of computing resources required for target processing of the first sensing data is large, a larger amount of computing resources is allocated to guarantee sufficient resources for target processing and obtaining high-quality second sensing data. Conversely, when the amount of computing resources required for target processing of the first sensing data is small, a smaller amount of computing resources is allocated, allowing surplus computing resources to be distributed to other sensing devices requiring greater computing power, thus avoiding waste.

[0151] In some embodiments, the processing unit 603 is further configured to allocate bandwidth to each sensing device based on the bandwidth required by each sensing device and the total bandwidth of the second optical interface 601.

[0152] In some embodiments, at least one optical signal is obtained by converting at least one compressed data, wherein the at least one compressed data is obtained by shallow compression of at least one first sensing data by at least one sensing device;

[0153] Conversion module 602 is used to convert at least one optical signal into at least one compressed data;

[0154] The processing unit 603 is also configured to decompress at least one compressed data to obtain at least one first sensing data.

[0155] The processing unit 603 is also configured to perform deep compression on at least one second sensing data obtained;

[0156] Shallow compression uses a compression ratio lower than the compression ratio threshold, while deep compression uses a compression ratio higher than the compression ratio threshold. The compression ratio used by the processing unit 603 for deep compression is greater than the compression ratio used by the sensing device for shallow compression.

[0157] The sensing device may include a small amount of computing resources to perform shallow compression on the first sensing data, which can reduce the amount of data in the optical signal and reduce the bandwidth resources required to transmit the optical signal.

[0158] In some embodiments, target processing includes one or more of the following: enhancement processing, noise reduction processing, target detection processing, target recognition processing, perceptual data fusion, or perceptual data stitching.

[0159] In some embodiments, the second optical interface 601 is a photoelectric composite interface, and the second optical interface 601 is also used to provide power to multiple sensing devices.

[0160] In some embodiments, the above-described communication-sensing integrated system is a PON-structured network or an FTTR-structured network.

[0161] The following will describe in detail the communication process between the sensing device and the processing device in a communication-sensing integrated system through any of the following embodiments.

[0162] Referring to Figure 8, this application embodiment provides a communication method 800, which is applied to the communication and sensing integrated system 100 shown in Figures 1, 2, or 3. In the communication method 800, the sensing device uses a dynamic compression ratio to compress the sensed data and sends the compressed sensed data to the processing device through a point-to-multipoint optical connection. The communication method 800 includes the following process.

[0163] Step 801: The first sensing device acquires the first sensing data.

[0164] In step 801, the first sensing device performs sensing to obtain first sensing data.

[0165] The first sensing device is any sensing device in the integrated communication and sensing system. The first sensing device senses its surrounding environment to obtain first sensing data.

[0166] In some embodiments, referring to FIG9, the first sensing device includes a sensing sensor, and the first sensing device obtains first sensing data by sensing its surrounding environment through the sensing sensor.

[0167] The first sensing data includes background data and may also include foreground data. The first sensing device obtains the third sensing data in the previous sensing and obtains the first sensing data in the current sensing; that is, the first sensing data is the data of the current frame, and the third sensing data is the data of the previous frame. Background data includes data that is the same between the third sensing data and the first sensing data, while foreground data includes data that is different between the third sensing data and the first sensing data.

[0168] For example, the first sensing device is a camera device, which includes an image sensor and a lens. The camera device currently exposes its surrounding environment using the image sensor to obtain first sensing data, which is first image data and represents the current frame image data. The camera device then exposes its surrounding environment again to obtain third sensing data, which is third image data and represents the previous frame image data.

[0169] If a moving object passes through the surrounding environment of the current camera device, the first image data includes a foreground image and a background image. The foreground image is the image of the moving object, and the background image is the image in the first image data excluding the image of the moving object. The foreground image is the foreground data, which includes the different data between the current frame image data and the previous frame image data, and the background image is the background data, which includes the same data between the current frame image data and the previous frame image data.

[0170] For example, the first sensing device is a radar device. The radar device's sensing sensors include a transmitter and a receiver. The transmitter is responsible for modulating and transmitting the signal (such as a millimeter-wave signal), and the receiver is responsible for receiving and demodulating the echo signal. The echo signal is the signal reflected back by an object from the transmitted signal. The radar device currently detects its surrounding environment through the transmitter and receiver, obtaining first sensing data, which is first point cloud data, representing the current frame's point cloud data. The radar device then detects its surrounding environment again, obtaining third sensing data, which is third point cloud data, representing the previous frame's point cloud data. Optionally, the radar device can be a millimeter-wave radar or a lidar, etc.

[0171] If a moving object passes through the surrounding environment of the radar device, the first point cloud data includes foreground data and background data. The foreground data is the point cloud data of the moving object, and the background data is the data in the first point cloud data excluding the point cloud data of the moving object. The foreground data includes the different data between the current frame point cloud data and the previous frame point cloud data, and the background data includes the same data between the current frame point cloud data and the previous frame point cloud data.

[0172] For example, the first sensing device is an audio acquisition device. The sensing sensors of the audio acquisition device include microphones and / or audio receivers. The audio acquisition device currently collects the sound of its surrounding environment through the microphone and / or audio receiver to obtain first sensing data, which is first audio data and is the audio data of the current frame. The audio acquisition device then collects the sound of its surrounding environment again to obtain third sensing data, which is third audio data and is the audio data of the previous frame.

[0173] If there is sound generated by an object in the surrounding environment of the current audio acquisition device, the first audio data includes foreground data and background data. The foreground data is the audio data generated by the object, and the background data is the data in the first audio data excluding the audio data generated by the object. The foreground data includes the different data between the current frame audio data and the previous frame audio data, and the background data includes the same data between the current frame audio data and the previous frame audio data.

[0174] If a communication and sensing integrated system includes multiple sensing devices, the other sensing devices besides the first sensing device also perform sensing and obtain sensing data, just like the first sensing device. For example, the communication and sensing integrated system also includes a second sensing device, which, like the first sensing device, senses its surrounding environment and obtains sensing data.

[0175] Step 802: The first sensing device acquires the target compression ratio, and compresses the first sensing data based on the target compression ratio to obtain the target compressed data.

[0176] In some embodiments, the target compression ratio includes a first compression ratio. The first compression ratio is the compression ratio used by the first sensing device to compress foreground data. Optionally, the target compression ratio may also include a second compression ratio, which is the compression ratio used by the first sensing device to compress background data, and the first compression ratio is smaller than the second compression ratio.

[0177] Optionally, the first compression ratio and / or the second compression ratio are pre-configured in the first sensing device, or the first compression ratio and / or the second compression ratio are determined based on the actual bandwidth available to the first sensing device. This allows for a smaller configured first compression ratio and / or second compression ratio when the available bandwidth resources of the point-to-multipoint optical connection are sufficient, to perform shallow compression on the first sensing data and retain more information. When the available bandwidth resources of the point-to-multipoint optical connection are insufficient, the configured first compression ratio and / or second compression ratio are larger, to perform deep compression on the first sensing data and reduce the bandwidth resources required to transmit the first sensing data.

[0178] In some embodiments, when the communication-sensing integrated system includes multiple sensing devices, the bandwidth actually available to the first sensing device is determined by the processing device based on the bandwidth required by each sensing device in the communication-sensing integrated system and the total bandwidth of the processing device's second optical interface.

[0179] The processing device can obtain the bandwidth required by each sensing device in the integrated communication and sensing system. Optionally, the processing device can obtain the bandwidth required by each sensing device in the following three ways.

[0180] In Method 1, the first sensing device counts the amount of sensing data it senses and needs to send, obtains the bandwidth it needs based on the counted data, sends the bandwidth it needs to the processing device, and the processing device receives the bandwidth required by the first sensing device.

[0181] In the integrated communication and sensing system, other sensing devices acquire the bandwidth they need, just like the first sensing device, and send the bandwidth they need to the processing device. The processing device receives the bandwidth from the other sensing devices.

[0182] Method 2: The first sensing device sends the sensed data to the processing device. The processing device receives the sensed data sent by the first sensing device, counts the amount of sensed data sent by the first sensing device, and obtains the bandwidth required by the first sensing device based on the counted amount of data.

[0183] In Method 2, the bandwidth that the first sensing device can actually use is not limited. That is, the first sensing device is allowed to send the sensing data obtained within a certain period of time without bandwidth limitation. The processing device counts the amount of sensing data sent by the first sensing device within this period of time and obtains the bandwidth required by the first sensing device based on the counted amount of data.

[0184] For other sensing devices in the integrated communication and sensing system, the processing device obtains the bandwidth required by the other sensing devices in the same way as described above.

[0185] After obtaining the bandwidth required by each sensing device in the integrated communication and sensing system, the processing device can allocate the actual usable bandwidth to each sensing device based on the total bandwidth of the second optical interface and the bandwidth required by each sensing device. Each sensing device is then notified of its actual usable bandwidth.

[0186] The bandwidth actually available to each sensing device is less than or equal to the bandwidth required by that sensing device. Optionally, the smaller the bandwidth required by a sensing device, the smaller the bandwidth that may be allocated to the sensing device in actual use; conversely, the larger the bandwidth required by a sensing device, the larger the bandwidth that may be allocated to the sensing device in actual use.

[0187] Method 3: The processing device allocates the bandwidth that the first sensing device can actually use based on the environment in which the first sensing device is located.

[0188] For example, if the environment where the first sensing device is located has a high volume of people, causing the foreground data in the sensing data perceived by the first sensing device to change frequently, the actual usable bandwidth allocated to the first sensing device will be larger. If the environment where the first sensing device is located has a low volume of people, causing the sensing data perceived by the first sensing device to not include foreground data, or to perceive only a small amount of sensing data including foreground data, the actual usable bandwidth allocated to the first sensing device will be smaller.

[0189] In some embodiments, after allocating the bandwidth that the first sensing device can actually use, the processing device can determine the first compression ratio and / or the second compression ratio of the first sensing device based on the bandwidth that the first sensing device can actually use, and notify the first sensing device of the first compression ratio and / or the second compression ratio.

[0190] In some embodiments, the first sensing device, upon receiving the bandwidth it can actually use, determines a first compression ratio and / or a second compression ratio based on the bandwidth it can actually use.

[0191] In some embodiments, the target compression ratio includes a third compression ratio. The third compression ratio is the compression ratio used by the first sensing device to compress the first sensing data.

[0192] Optionally, the third compression ratio is pre-configured in the first sensing device, or the third compression ratio is determined based on the actual bandwidth available to the first sensing device. This allows for a smaller configured third compression ratio when the available bandwidth resources of the point-to-multipoint optical connection are sufficient, enabling shallow compression of the first sensing data to retain more information. Conversely, when the available bandwidth resources of the point-to-multipoint optical connection are insufficient, a larger configured third compression ratio enables deep compression of the first sensing data, reducing the bandwidth resources required to transmit the first sensing data.

[0193] In step 802, the first sensing device may use the following methods to compress the first sensing data based on the target compression ratio to obtain the target compressed data.

[0194] In the first compression method, the first sensing device compresses the foreground data included in the first sensing data based on a first compression ratio to obtain first compressed data, and compresses the background data included in the first sensing data based on a second compression ratio to obtain second compressed data. The target compressed data includes the first compressed data and the second compressed data.

[0195] Foreground data consists of data that changes between the first and third sensing data, while background data consists of data that is the same between the first and third sensing data. Therefore, the first compression ratio is lower than the second compression ratio. This reduces information loss during foreground data compression, allowing the first compressed data to retain as much information as possible from the foreground data. Similarly, when compressing background data, the amount of data in the second compressed data can be minimized. By using both the first and second compression ratios, the amount of data in the target compressed data can be minimized while preserving as much useful information as possible. This reduces the bandwidth consumption of point-to-multipoint optical connections, enabling the integrated communication and sensing system to simultaneously include more sensing devices connected to the processing device.

[0196] Optionally, if the first perception data is first image data or first point cloud data, the target compression data may also include the position of the foreground data in the background data.

[0197] For example, the first perception data is the first image data, the foreground data of which is the image of the moving object, i.e., the foreground image, the background data is the background image in the first image data, and the position in the target compressed data is the position of the foreground image in the background image.

[0198] In the second compression method, the first sensing device compresses the foreground data included in the first sensing data based on the first compression ratio to obtain the first compressed data, and the target compressed data includes the first compressed data.

[0199] Before receiving the first sensing data, the first sensing device sends background data to the processing device, which includes the background data. In the second compression method, foreground data can be compressed, and the target compressed data includes the first compressed data. That is, the first sensing device can send the foreground data included in the first sensing data to the processing device, but not the background data included in the first sensing data. This reduces the bandwidth consumption of point-to-multipoint optical connections, allowing the integrated communication and sensing system to simultaneously include more sensing devices connected to the processing device.

[0200] Optionally, if the first perception data is first image data or first point cloud data, the target compression data may also include the position of the foreground data in the background data.

[0201] The third compression method involves the first sensing device compressing the first sensing data based on a third compression ratio to obtain third compressed data, and the target compressed data includes the third compressed data.

[0202] The third compression method is a shallow compression method. The third compression ratio is small, which can reduce the consumption of computing resources of the first sensing device when compressing the first sensing data.

[0203] In some embodiments, the target compressed data further includes a sensing timestamp of the first sensing data, which is the timestamp at which the first sensing device senses the first sensing data.

[0204] Step 803: The first sensing device sends the target compressed data to the processing device through a point-to-multipoint optical connection.

[0205] In step 803, the first sensing device converts the target compressed data into an optical signal and sends the optical signal to the processing device through the first optical interface included in the first sensing device on a point-to-multipoint optical connection to realize the transmission of the target compressed data.

[0206] Referring to Figure 9, the first sensing device also includes a first conversion module. The first sensing device converts the target compressed data into an optical signal through the first conversion module, and sends the optical signal to the processing device through the first optical interface included in the first sensing device on a point-to-multipoint optical connection.

[0207] In some embodiments, the first sensing device converts the target compressed data into a target signal in a target communication protocol defined format through a first conversion module, and converts the target signal into an optical signal. The target communication protocol is the communication protocol used between the first sensing device and the processing device.

[0208] Optionally, the first conversion module can be an electro-optical conversion module, etc., and the sensing sensor can be located in the sensing module.

[0209] In some embodiments, the target compressed data includes first compressed data and second compressed data. The first sensing device converts the first compressed data and second compressed data into optical signals and sends the optical signals to the processing device via a first optical interface included in the first sensing device on a point-to-multipoint optical connection to realize the transmission of the first compressed data and second compressed data.

[0210] In some embodiments, the target compressed data includes first compressed data. The first sensing device converts the first compressed data into an optical signal and sends the optical signal to the processing device over a point-to-multipoint optical connection through a first optical interface included in the first sensing device, thereby realizing the transmission of the first compressed data.

[0211] In some embodiments, the target compressed data includes third compressed data. The first sensing device converts the third compressed data into an optical signal and sends the optical signal to the processing device over a point-to-multipoint optical connection through a first optical interface included in the first sensing device, thereby realizing the transmission of the third compressed data.

[0212] In a communication and sensing integrated system that includes multiple sensing devices, the other sensing devices, in addition to the first sensing device, use point-to-multipoint optical connections to send target compressed data to the processing device, just like the first sensing device.

[0213] Step 804: The processing device receives the target compressed data, decompresses the target compressed data, and obtains the first sensing data.

[0214] In some embodiments, referring to FIG10, the processing device includes a second conversion module and a computing resource pool in addition to the second optical interface. The processing device receives the optical signal sent by the first sensing device from the point-to-multipoint optical connection through the second optical interface, converts the optical signal into target compressed data through the second conversion module, and then decompresses the target compressed data through the operation in step 804 to obtain the first sensing data.

[0215] Optionally, the second conversion module can be a photoelectric conversion module, etc.

[0216] In some embodiments, the target compressed data includes first compressed data and second compressed data. The processing device decompresses the first compressed data to obtain foreground data of the first sensing data, and decompresses the second compressed data to obtain background data of the first sensing data. Based on the foreground data and background data of the first sensing data, the first sensing data is obtained.

[0217] Optionally, if the first sensing data is first image data or first point cloud data, the target compressed data also includes the position of the foreground data in the background data. Based on the position of the foreground data in the background data, the foreground data and background data of the first sensing data are fused to obtain the first sensing data.

[0218] For example, the first perceived data is the first image data, and the position in the target compressed data is the position of the foreground image in the background image. The processing device decompresses the first compressed data to obtain the foreground image of the first image data, and decompresses the second compressed data to obtain the background image of the first image data. Based on the position of the foreground image in the background image, the foreground image and the background image of the first image data are fused to obtain the first image data.

[0219] In some embodiments, the target compressed data includes first compressed data, and the processing device includes background data. The processing device decompresses the first compressed data to obtain foreground data of the first sensing data, and obtains the first sensing data based on the background data and the foreground data of the first sensing data.

[0220] Optionally, if the first sensing data is first image data or first point cloud data, the target compressed data also includes the position of the foreground data in the background data. Based on the position of the foreground data in the background data, the foreground data of the background data and the first sensing data are fused to obtain the first sensing data.

[0221] The background data included in the processing device is from the data sent by the first sensing device prior to the current event. For example, when the first sensing device first sends sensing data, the processing device receives the first sent sensing data, obtains the background data from the first sent sensing data, and saves the background data.

[0222] In some embodiments, the target compressed data includes third compressed data, and the processing device decompresses the third compressed data to obtain first sensing data.

[0223] When the processing device decompresses the first sensing data, it may also decompress the sensing timestamp of the first sensing data.

[0224] In a communication and sensing integrated system comprising multiple sensing devices, for sensing devices other than the first sensing device, the processing device obtains the first sensing data of the other sensing devices according to the operation in step 804. Optionally, the sensing timestamp of the first sensing data of the other sensing devices can also be obtained.

[0225] Step 805: The processing device performs target processing on the first sensing data to obtain the second sensing data.

[0226] Target processing includes one or more of the following: enhancement processing, noise reduction processing, target detection processing, target recognition processing, perceptual data fusion, or perceptual data stitching.

[0227] The quality of the second sensing data is higher than that of the first sensing data, and / or, the second sensing data contains target boxes that include detected targets, and / or, the second sensing data includes the categories of identified targets, and / or, the size of the second sensing data is larger than that of the first sensing data, and / or, the dimensionality of the second sensing data is higher than that of the first sensing data, etc.

[0228] Referring to Figure 10, the processing device includes a computing resource pool. Based on the amount of foreground data from the first sensing device, the processing device allocates computing resources from the pool to the first sensing data. Based on the allocated computing resources, the first sensing data undergoes target processing to obtain the second sensing data.

[0229] For example, the first sensing device is a camera, the first sensing data is the first image data, and the foreground data of the first image data is the foreground image. The amount of data in the foreground image can be represented by the number of moving object images included in the foreground image. That is, the processing device allocates computing resources from the computing resource pool to the first image data based on the number of moving object images, and performs target processing on the first image data based on the allocated computing resources.

[0230] The computing resources allocated in this way change dynamically with the amount of foreground data in the first perception data. This way, the quality of processing the first perception data will not be reduced due to insufficient computing resources, so that low-quality second perception data will not be obtained. At the same time, excessive computing resources will not be allocated, which would lead to a waste of computing resources.

[0231] In some embodiments, the target processing includes enhancement processing and / or denoising processing. The processing device performs enhancement processing and / or denoising processing on the first sensing data to obtain second sensing data, such that the quality of the second sensing data is higher than that of the first sensing data.

[0232] For example, the first perceived data is the first image data. The processing device performs enhancement processing and / or noise reduction processing on the first image data to obtain the second image data, so that the quality of the second image data is higher than that of the first image data.

[0233] For example, the first sensing data is the first point cloud data. The processing device performs enhancement processing and / or noise reduction processing on the first point cloud data to obtain the second point cloud data, so that the quality of the second point cloud data is higher than that of the first point cloud data.

[0234] For example, the first perceived data is the first audio data, and the processing device performs enhancement processing and / or noise reduction processing on the first audio data to obtain the second audio data, such that the quality of the second audio data is higher than that of the first audio data.

[0235] In some embodiments, target processing includes target detection processing, which involves detecting whether a target exists in the first sensing data. If a target is present in the first sensing data, a target bounding box can be used to include the target. Therefore, the processing device performs target detection on the first sensing data. If a target is detected in the first sensing data, a target bounding box including the target is added to the first sensing data to obtain the second sensing data.

[0236] For example, the first perception data is the first image data. The processing device performs target detection on the first image data. If the first image data is found to contain a target, a target bounding box containing the target is added to the first image data to obtain the second image data.

[0237] For example, if the target is a vehicle, the target detection process can detect whether the first image data contains a vehicle image. If the first image data is found to contain a vehicle image, a bounding box containing the vehicle image can be added to the first image data.

[0238] For example, if the target is fruit, target detection processing can detect whether the first image data contains a fruit image. If the first image data is found to contain a fruit image, a target bounding box containing the fruit image can be added to the first image data.

[0239] The target can also be other items, which will not be listed here.

[0240] In some embodiments, target processing includes target recognition processing, which involves identifying the categories of targets included in the first sensing data, i.e., identifying which categories of targets are included in the first sensing data. Therefore, the processing device performs target recognition on the first sensing data; if the first sensing data is identified as containing targets, the target categories are added to the first sensing data to obtain second sensing data.

[0241] For example, the first perception data is the first image data. The processing device performs target recognition on the first image data. If the first image data is identified to include a target, the target category is added to the first image data to obtain the second image data, so that the target can be labeled using the category in the second image data.

[0242] For example, if the target is a vehicle, the first image data includes vehicle images. Through target recognition processing, it can be detected that the first image data contains vehicle images, and the category of the target included in the first image data can be identified as a vehicle, thus adding the target category to the first image data.

[0243] For example, if the target is fruit, the first image data includes at least one fruit image, which includes an apple image and a peach image. Target recognition processing can detect that the first image data includes apple and peach images, and identify the target categories as apples and peaches, thus adding the target category to the first image data.

[0244] The target can also be other items, which will not be listed here.

[0245] Target recognition processing identifies the type of target contained in the input perceptual data. Its input is perceptual data (such as an image), and its output is the category to which the target in the perceptual data (image) belongs. Target detection processing, in addition to indicating the type of target in the input perceptual data (image), also outlines the specific location of the target.

[0246] In some embodiments, target processing includes sensing data stitching, which refers to stitching together multiple sensing data of the same type into a single sensing data of a larger size. The integrated communication and sensing system includes multiple sensing devices of the same type, including a first sensing device. A processing device acquires multiple first sensing data with the same sensing timestamp. These multiple first sensing data are obtained by the multiple sensing devices at the same sensing timestamp and are of the same type. The multiple first sensing data are then stitched together to obtain second sensing data, such that the size of the second sensing data is larger than the size of each individual first sensing data.

[0247] Because multiple sensing and processing devices in a communication and sensing integrated system are precisely clock-synchronized through point-to-multipoint optical connections, multiple first sensing data with the same sensing timestamp are data sensed by multiple sensing devices at the same time.

[0248] For example, the first sensing data is first image data. The integrated communication and sensing system includes two camera devices. The processing device acquires two first image data sets with the same sensing timestamp. These two first image data sets are obtained by the two camera devices exposing at the same sensing timestamp. The processing device stitches these two first image data sets together to obtain second image data. The size of the second image data set is larger than the size of each first image data set.

[0249] For example, the first sensing data is the first point cloud data. The integrated communication and sensing system includes three radar devices. The processing device acquires three first point cloud data sets with the same sensing timestamp. These three first point cloud data sets were detected by the three radar devices at the same sensing timestamp. The processing device stitches these three first point cloud data sets together to obtain the second point cloud data set. The size of the second point cloud data set is larger than the size of each individual first point cloud data set.

[0250] In some embodiments, target processing includes sensing data fusion, which refers to merging multiple sensing data of different types into a single sensing data. The type of the merged sensing data may differ from the types of the individual sensing data. The integrated communication and sensing system includes multiple sensing devices, which may be of different types, including a first sensing device. A processing device acquires multiple first sensing data with the same sensing timestamp. These multiple first sensing data are obtained by the multiple sensing devices at the same sensing timestamp and are of different types. The multiple first sensing data are then fused to obtain second sensing data, such that the dimension of the second sensing data is higher than the dimension of each individual first sensing data.

[0251] For example, a communication-sensing integrated system includes two sensing devices: one is a camera, and the other is an audio acquisition device. A processing device acquires two sets of first sensing data with the same sensing timestamp. One set is the first image data obtained by the camera, and the other is the first audio data acquired by the audio acquisition device. The sensing timestamp of the first image data (also called the exposure timestamp) and the sensing timestamp of the first audio data (also called the acquisition timestamp) are the same. The processing device fuses the first image data and the first audio data to obtain second sensing data. The second sensing data is audio and video data, including both audio and video dimensions, making the second sensing data have a higher dimension than each set of first sensing data.

[0252] For example, a communication-sensing integrated system includes two sensing devices: a radar device and an audio acquisition device. The processing device acquires two sets of first sensing data with the same sensing timestamp. One set is the first point cloud data detected by the radar device, and the other is the first audio data acquired by the audio acquisition device. The sensing timestamps of the radar device detecting the first point cloud data (also called the detection timestamp) and the audio acquisition device acquiring the first audio data (also called the acquisition timestamp) are identical. The processing device fuses the first point cloud data and the first audio data to obtain second sensing data. The second sensing data is audio and video data, including both audio and video dimensions, making the second sensing data more dimensional than each set of first sensing data.

[0253] In some embodiments, after the target processing device performs enhancement processing and / or denoising processing on the first sensing data, it performs target detection processing, target recognition processing, sensing data fusion and / or sensing data stitching on the enhanced and / or denoised first sensing data to obtain second sensing data.

[0254] In some embodiments, target processing may also include other processing, such as collaborative processing. For the first sensing data from the first sensing device, if an event is detected based on the first sensing data, the sensing timestamp of the first sensing data is obtained, sensing data sensed at that sensing timestamp is obtained from the sensing data of other sensing devices, and the obtained sensing data is provided to the user.

[0255] For example, the first sensing device is an audio acquisition device, and the first sensing data is audio data. Other sensing devices include camera devices, and the sensing data perceived by the camera devices is image data. If an event is detected based on the audio data, the event may be an abnormal sound (such as a falling sound or a shouting sound). When an abnormal sound is detected based on the audio data, the sensing timestamp of the audio data is obtained. The image data perceived at that sensing timestamp is then obtained from the image data perceived by the camera devices, and the obtained image data is provided to the user to demonstrate the event that occurred.

[0256] Besides collaborative processing, other processing methods included in target processing will not be listed and explained one by one.

[0257] In some embodiments, after obtaining the second sensing data, the processing device can also compress the second sensing data based on a fourth compression ratio to obtain fourth compressed data. The fourth compression ratio is greater than the third compression ratio, so compressing the second sensing data based on the fourth compression ratio is deep compression. Compared with shallow compression, deep compression requires more computing resources.

[0258] Optionally, the processing device may also send fourth compressed data, for example, the processing device may send fourth compressed data to a storage device, and the storage device may store the fourth compressed data.

[0259] In this embodiment, a first sensing device senses first sensing data, compresses the first sensing data based on a target compression ratio to obtain target compressed data, and sends the target compressed data to a processing device via a point-to-multipoint optical connection. The processing device decompresses the target compressed data to obtain the first sensing data, and performs target processing on the first sensing data to obtain second sensing data. Since the processing device performs target processing on the first sensing data, the first sensing device itself does not need to perform target processing; that is, the first sensing device does not need to include computing resources for target processing, thereby reducing the cost of the first sensing device. Since the processing device includes sufficient computing resources, sufficient computing resources can be allocated to the first sensing data during target processing, maintaining the quality of the processed first sensing data and obtaining high-quality second sensing data. In other words, allocating computing resources for the first sensing data through the processing device avoids wasting computing resources and does not reduce the quality of the second sensing data. The target compression ratio is determined based on the actual bandwidth available to the first sensing device, ensuring that the bandwidth resources actually available to the first sensing device meet the bandwidth required to send the first sensing data, enabling more sensing devices in the integrated communication and sensing system to connect to the processing device.

[0260] Referring to Figure 11, this application embodiment provides a communication method 1100, which is applied to the communication and sensing integrated system 100 shown in Figures 1, 2, or 3. In the communication method 1100, when a first sensing device senses first sensing data, it sends the first sensing data to a processing device via a point-to-multipoint optical connection. The sensing data previously sent by the first sensing device before sending the first sensing data is the fourth sensing data, and there is an interval of at least one frame of fifth sensing data between the first sensing data and the fourth sensing data. The communication method 1100 includes the following process.

[0261] Step 1101: Same as step 801, and will not be described in detail here.

[0262] Step 1102: When the first sensing device determines to send the first sensing data, it obtains the target compression ratio, compresses the first sensing data based on the target compression ratio, and obtains the target compressed data.

[0263] In some embodiments, the first sensing device acquires the number of sensing data points between the first sensing data and the fourth sensing data, where the fourth sensing data is the sensing data previously sent by the first sensing device to the processing device. When this number reaches a threshold, it is determined to send the first sensing data, where the threshold is an integer greater than or equal to 1. For ease of explanation, the interval sensing data will be referred to as the fifth sensing data.

[0264] Optionally, the number threshold includes a first number threshold and a second number threshold, wherein the first number threshold is less than the second number threshold.

[0265] When the initial sensing data includes foreground data, the first sensing device will continuously send N frames of the sensed data, where N is an integer greater than 1. Afterward, the first sensing device will send the sensed data at least once every frame. Even when the sensing data does not include foreground data, the first sensing device will still send the sensed data at least once every frame.

[0266] Optionally, the operation of the first sensing device determining to send the first sensing data can be:

[0267] If both the first and fourth sensing data include foreground data, and the fourth sensing data is the sensing data of the Nth frame that includes foreground data, or the fourth sensing data is the sensing data perceived after the Nth frame, the first sensing device obtains the number of fifth sensing data intervals between the first and fourth sensing data. When the number reaches a first threshold, it determines to send the first sensing data. If the fourth sensing data includes foreground data, and the first sensing data is the sensing data of the last frame that includes foreground data, then the first sensing device can determine to send the first sensing data.

[0268] If neither the first sensing data nor the fourth sensing data includes foreground data, the first sensing device obtains the number of fifth sensing data points between the first sensing data and the fourth sensing data. When the number reaches the second threshold, it determines to send the first sensing data.

[0269] If the fourth sensing data does not include foreground data, and the first sensing data is the first frame that includes foreground data, then the first sensing device determines to send the first sensing data. The first sensing data is the sensing data of the first frame that includes foreground data, and the sensing data of the next N-1 consecutive frames that include foreground data also need to be sent.

[0270] In this transmission method, since the first number threshold is less than the second number threshold, the frequency of transmitting sensing data when foreground data is included is greater than the frequency of transmitting sensing data when foreground data is not included.

[0271] In some embodiments, the first sensing device determines whether the first sensing data is keyframe sensing data. If the first sensing data is keyframe sensing data, it determines to send the first sensing data. That is, the first sensing device sends keyframe sensing data to the processing device, but may not send non-keyframe sensing data other than keyframe sensing data. In other words, both the fourth sensing data and the first sensing data are keyframe sensing data.

[0272] For example, if the first perception data is the first image data, and if the first image data is keyframe image data, then the first image data is determined to be sent.

[0273] The processing device can schedule the timing of the first sensing device sending its first sensing data. For example, if the current time is the peak time for sensing devices in a communication-sensing integrated system to send sensing data, and a large number of sensing devices in the system need to send sensing data, but the total bandwidth of the processing device's second optical interface is limited, allowing a large number of sensing devices to send sensing data simultaneously could lead to congestion in the point-to-multipoint optical connection. To avoid this congestion, the processing device can schedule the timing of each sensing device's data transmission, allowing some devices to send data first and others later, thus achieving staggered transmission.

[0274] Optionally, in implementation: the first sensing device includes a buffer. After the processing device schedules the first sensing device to sense the first sensing data, it first buffers the first sensing data in the buffer for a period of time. Then, when the first sensing device senses the first sensing data and determines to send it, it first buffers the first sensing data in the buffer. The length of this period is equal to a duration threshold. The duration threshold may be a threshold assigned to the first sensing device by the processing device, a threshold pre-configured in the first sensing device, or a threshold randomly generated by the first sensing device. If the buffering time of the first sensing data reaches the duration threshold, the first sensing device performs the operation of sending the first sensing data; that is, the first sensing device obtains the target compression ratio, compresses the first sensing data based on the target compression ratio, and obtains the target compressed data to be sent. Alternatively,

[0275] After the processing device schedules the first sensing device to sense the first sensing data, it does not need to cache the first sensing data in the buffer for a period of time. Instead, when the first sensing device senses the first sensing data and determines to send it, it performs the operation of sending the first sensing data. That is, the first sensing device obtains the target compression ratio, compresses the first sensing data based on the target compression ratio, and obtains the target compressed data to be sent.

[0276] For other sensing devices in the integrated communication and sensing system, the processing device schedules the other sensing devices in the same way as described above. This allows a large number of sensing devices in the integrated communication and sensing system to send sensing data at off-peak times, avoiding congestion in point-to-multipoint optical connections, so that more sensing devices can be connected to the processing device.

[0277] In some embodiments, the first sensing device determines to send the first sensing data if the first sensing data includes foreground data. This enables the transmission of the sensed data when sensing data including foreground data is sensed, and the transmission of the sensed data not including foreground data when sensing data not including foreground data is not realized.

[0278] Steps 1103-1104 are the same as steps 803-804 shown in Figure 8, and will not be described in detail here.

[0279] Step 1105: Based on the changing pattern of the foreground data in the fourth sensing data, the first sensing data, and the fourth sensing data, the processing device acquires at least one frame of fifth sensing data located between the fourth sensing data and the first sensing data.

[0280] The processing device includes fourth sensing data, which is the data previously received from the first sensing device.

[0281] In this process, when the first sensing device first senses sensing data including foreground data, it continuously sends the sensed data N times. That is, the processing device receives N frames of sensing data and, based on the foreground data included in these N frames, determines the variation pattern of the foreground data. Thus, when the fourth sensing data is received, based on the determined variation pattern of the foreground data and the foreground data included in the fourth sensing data, the variation pattern of the foreground data in the fourth sensing data is obtained.

[0282] In step 1105, the processing device, based on the variation pattern of the foreground data in the fourth sensing data, the first sensing data, and the fourth sensing data, uses a frame interpolation algorithm to obtain at least one frame of fifth sensing data located between the fourth sensing data and the first sensing data. This method, based on the variation pattern of the foreground data, allows at least one frame of fifth sensing data to more closely approximate the data actually perceived by the first sensing device, thus improving the accuracy of the at least one frame of fifth sensing data.

[0283] In some embodiments, the processing device further acquires the perception time frame of each frame of fifth perception data in at least one frame of fifth perception data.

[0284] In some embodiments, the processing device may further obtain the change pattern of the foreground data in the first sensing data based on the change pattern of the foreground data in the fourth sensing data, the foreground data in at least one frame of the fifth sensing data, and the foreground data in the first sensing data. This allows the processing device to, upon receiving sensing data from the first sensing device again, acquire at least one frame of sensing data located between the first sensing data and the received sensing data, based on the change pattern of the foreground data in the first sensing data, the first sensing data, and the received sensing data. This dynamically adjusts the change pattern of the foreground data stored locally by the processing device, thereby improving the accuracy of acquiring at least one frame of sensing data.

[0285] Optionally, assuming the first perceived data is the first image data, the change pattern of the foreground data is the change pattern of the moving object. The change pattern of the moving object includes one or more of the following: information such as the moving object's speed, direction of motion, or posture.

[0286] Step 1106: The processing device performs target processing on the first sensing data to obtain the second sensing data, and performs target processing on at least one frame of the fifth sensing data to obtain at least one frame of the sixth sensing data.

[0287] The detailed process of the processing device performing target processing on the first sensing data to obtain the second sensing data can be found in step 805 of method 800 shown in Figure 8, and will not be described in detail here.

[0288] The detailed implementation process of the processing device for target processing of each frame of fifth sensing data is the same as the detailed implementation process for target processing of the first sensing data. You can also refer to the relevant content of step 805 of method 800 shown in Figure 8, which will not be described in detail here.

[0289] In this embodiment, a first sensing device senses first sensing data, with at least one frame of sensing data remaining between the first sensing data and the previously transmitted fourth sensing data. The first sensing data is compressed based on a target compression ratio to obtain target compressed data, which is then sent to a processing device via a point-to-multipoint optical connection. The processing device decompresses the target compressed data to obtain the first sensing data and performs target processing on it to obtain second sensing data. Since the processing device performs target processing on the first sensing data, the first sensing device itself does not need to perform target processing, meaning it does not require computing resources for target processing, thus reducing its cost. Because the processing device includes sufficient computing resources, it can allocate adequate resources to the first sensing data during target processing, maintaining the quality of the processed first sensing data and obtaining high-quality second sensing data. In other words, allocating computing resources to the first sensing data through the processing device avoids wasting computing resources and does not degrade the quality of the second sensing data. The target compression ratio is determined based on the actual bandwidth available to the first sensing device, ensuring that the available bandwidth resources meet the bandwidth requirements for transmitting the first sensing data. This allows more sensing devices in the integrated communication and sensing system to connect to the processing device. Furthermore, there is at least one frame of sensing data between the first sensing data and the previously transmitted fourth sensing data. This means the first sensing device does not need to transmit data every time it senses it, reducing the amount of sensing data transmitted. This reduces the bandwidth consumption of point-to-multipoint optical connections, allowing more sensing devices in the integrated communication and sensing system to connect to the processing device. Moreover, since the processing device can acquire at least one frame of fifth sensing data based on the changing patterns of the foreground data in the fourth sensing data, as well as the first and fourth sensing data, it will not affect the services of the first sensing device.

[0290] Referring to Figure 12, this application embodiment provides a communication sensing integrated system 1200. The communication sensing integrated system 1200 can be the communication sensing integrated system 100 shown in Figure 1, Figure 2 or Figure 3, or the communication sensing integrated system 1200 can be the communication sensing integrated system applied by method 800 shown in Figure 8 or method 1100 shown in Figure 11.

[0291] The communication and sensing integrated system 1200 includes a processing device 1201 and multiple sensing devices 1202. For each sensing device 1202, the sensing device 1202 includes a first optical interface, and the processing device 1201 includes a second optical interface. The first optical interface of the sensing device 1202 is connected to the second optical interface of the processing device 1201 through a point-to-multipoint optical connection.

[0292] Sensing device 1202 is used to acquire first sensing data;

[0293] The sensing device 1202 is also used to send first sensing data to the processing device 1201 via an optical connection;

[0294] The processing device 1201 is used to perform target processing on the first sensing data to obtain the second sensing data.

[0295] Optionally, the detailed implementation process of the sensing device 1202 acquiring the first sensing data can be found in step 801 of method 800 shown in Figure 8 or step 1101 of method 1100 shown in Figure 11, and will not be described in detail here.

[0296] Optionally, the detailed implementation process of the sensing device 1202 sending the first sensing data to the processing device 1201 can be found in step 803 of method 800 shown in Figure 8 or step 1103 of method 1100 shown in Figure 11, and will not be described in detail here.

[0297] Optionally, the processing device 1201 is used to perform target processing on the first sensing data to obtain the second sensing data. For the detailed implementation process, please refer to the relevant content in step 805 of method 800 shown in Figure 8 or step 1106 of method 1100 shown in Figure 11, which will not be described in detail here.

[0298] Optionally, the first sensing data includes background data and foreground data. The background data is the same as the third sensing data and the first sensing data. The foreground data is the different data between the third sensing data and the first sensing data. The third sensing data is the previous frame data acquired by the sensing device 1202.

[0299] Optionally, the sensing device 1202 is used to compress foreground data based on a first compression ratio to obtain first compressed data, and compress background data based on a second compression ratio to obtain second compressed data, wherein the first compression ratio is less than the second compression ratio; and to send the first compressed data and the second compressed data to the processing device 1201 through the optical connection.

[0300] The processing device 1201 is also used to decompress the first compressed data to obtain foreground data, and to decompress the second compressed data to obtain background data; and to obtain the first sensing data based on the foreground data and the background data.

[0301] Optionally, the detailed implementation process of the sensing device 1202 obtaining first compressed data by compressing foreground data based on a first compression ratio and obtaining second compressed data by compressing background data based on a second compression ratio can be found in step 802 of method 800 shown in Figure 8 or step 1102 of method 1100 shown in Figure 11, and will not be described in detail here.

[0302] Optionally, the detailed implementation process of the sensing device 1202 sending the first compressed data and the second compressed data to the processing device 1201 can be found in step 803 of method 800 shown in Figure 8 or step 1103 of method 1100 shown in Figure 11, and will not be described in detail here.

[0303] Optionally, the processing device 1201 decompresses the first compressed data to obtain foreground data, decompresses the second compressed data to obtain background data, and obtains the first sensing data based on the foreground data and background data. For details of the implementation process, please refer to step 804 of method 800 shown in Figure 8 or step 1104 of method 1100 shown in Figure 11, which will not be described in detail here.

[0304] Optionally, the processing device 1201 includes background data;

[0305] The sensing device 1202 is used to compress foreground data based on a first compression ratio to obtain first compressed data; and to send the first compressed data to the processing device through the optical connection.

[0306] The processing device 1201 is also used to decompress the first compressed data to obtain foreground data; and to obtain the first sensing data based on the foreground data and the background data.

[0307] Optionally, the detailed implementation process of the sensing device 1202 obtaining the first compressed data based on the first compression ratio is described in step 802 of method 800 shown in Figure 8 or step 1102 of method 1100 shown in Figure 11, and will not be described in detail here.

[0308] Optionally, the detailed implementation process of the sensing device 1202 sending the first compressed data to the processing device 1201 can be found in step 803 of method 800 shown in Figure 8 or step 1103 of method 1100 shown in Figure 11, and will not be described in detail here.

[0309] Optionally, the detailed implementation process of the processing device 1201 decompressing the first compressed data to obtain foreground data, and obtaining the first sensing data based on the foreground data and background data, can be found in step 804 of method 800 shown in Figure 8 or step 1104 of method 1100 shown in Figure 11, and will not be described in detail here.

[0310] Optionally, the first compression ratio is determined based on the bandwidth that the sensing device 1202 can actually use.

[0311] Optionally, the processing device 1201 includes fourth sensing data, which is the data previously received by the processing device 1201, and there is at least one frame of sensing data between the first sensing data and the fourth sensing data.

[0312] The processing device 1201 is also used to acquire at least one frame of fifth sensing data located between the fourth sensing data and the first sensing data based on the changing pattern of the foreground data in the fourth sensing data, the first sensing data, and the fourth sensing data; and to perform target processing on the at least one frame of fifth sensing data.

[0313] Optionally, the detailed implementation process of the processing device 1201 acquiring at least one frame of fifth sensing data located between the fourth sensing data and the first sensing data based on the changing pattern of the foreground data in the fourth sensing data, the first sensing data, and the fourth sensing data is shown in step 1105 of method 1100 in Figure 11, and will not be described in detail here.

[0314] Optionally, the detailed implementation process of the processing device 1201 performing target processing on at least one frame of fifth sensing data is described in step 1106 of method 1100 shown in Figure 11, and will not be described in detail here.

[0315] Optionally, the processing device 1201 is used for:

[0316] Allocate computing resources based on the amount of foreground data;

[0317] Based on the allocated computing resources, the first sensing data is processed to obtain the second sensing data.

[0318] Optionally, the detailed implementation process of the processing device 1201 allocating computing resources based on the amount of foreground data can be found in step 805 of method 800 shown in Figure 8, and will not be described in detail here.

[0319] Optionally, the detailed implementation process of the processing device 1201 performing target processing on the first sensing data based on the allocated computing resources can be found in step 805 of method 800 shown in Figure 8, and will not be described in detail here.

[0320] Optionally, the processing device 1201 is also used to allocate bandwidth to each sensing device based on the bandwidth required by each sensing device and the total bandwidth of the second optical interface.

[0321] Optionally, the processing device 1201 allocates bandwidth to each sensing device based on the bandwidth required by each sensing device and the total bandwidth of the second optical interface. For details of the implementation process, please refer to the relevant content in step 802 of method 800 shown in Figure 8, which will not be described in detail here.

[0322] Optionally, target processing includes one or more of the following: enhancement processing, noise reduction processing, target detection processing, target recognition processing, perceptual data fusion, or perceptual data stitching.

[0323] Optionally, the sensing device is a camera device, and the first sensing data is image data; or,

[0324] The sensing device is a radar device, and the initial sensing data is point cloud data; or...

[0325] The sensing device is an audio acquisition device, and the first sensing data is audio data.

[0326] Optionally, the sensing device is a camera device, which includes a lens and an image sensor.

[0327] Optionally, the notification and sensing integrated system is a PON-structured network or an FTTR-structured network.

[0328] In this embodiment, the sensing device sends first sensing data to the processing device via a point-to-multipoint optical connection. The processing device performs target processing on the first sensing data to obtain second sensing data, thereby migrating the target processing from the sensing device to the processing device. The processing device has sufficient computing resources to perform target processing on the first sensing data, resulting in high-quality second sensing data. Since the sensing device does not perform target processing, there is no need to deploy computing resources for target processing on the sensing device. This not only reduces the cost of the sensing device but also improves the quality of the second sensing data, achieving both cost reduction and improved sensing data quality.

[0329] Referring to Figure 13, this application embodiment provides a computing device 1300. For example, the computing device 1300 may be a sensing device or a processing device in the communication and sensing integrated system 100 shown in Figures 1, 2, or 3.

[0330] As shown in Figure 13, the computing device 1300 includes a bus 1302, a processor 1304, a memory 1306, and a communication interface 1308. The processor 1304, the memory 1306, and the communication interface 1308 communicate with each other via the bus 1302. The computing device 1300 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1300.

[0331] Bus 1302 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 13, but this does not imply that there is only one bus or one type of bus. Bus 1302 can include pathways for transmitting information between various components of computing device 1300 (e.g., processor 1304, memory 1306, communication interface 1308).

[0332] The processor 1304 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0333] The memory 1306 may include volatile memory, such as random access memory (RAM). The memory 1306 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0334] Referring to Figure 13, the memory 1306 stores executable program code, which includes a processing module and a sensing module. The processor 1304 executes the processing module and sensing module included in the executable program code to respectively implement the functions of the processing device 1201 and sensing device 1202 in the system 100 shown in Figure 12, thereby implementing the method provided in any of the above embodiments. That is, the memory 1306 stores instructions for executing the method provided in any of the above embodiments. Alternatively,

[0335] The communication interface 1308 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1300 and other devices or communication networks.

[0336] This application also provides a communication-sensing integrated cluster. The communication-sensing integrated cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a desktop computer, a laptop computer, or a smartphone, or other terminal device.

[0337] As shown in Figure 14, the communication-sensing integrated cluster includes at least one computing device 1300. The memory 1306 of one or more computing devices 1300 in the communication-sensing integrated cluster may store the same instructions for executing the methods provided in any of the above embodiments.

[0338] In some possible implementations, the memory 1306 of one or more computing devices 1300 in the integrated communication and sensing cluster may also store partial instructions for executing the above-described communication methods. In other words, a combination of one or more computing devices 1300 can jointly execute instructions for performing the methods provided in any of the above embodiments.

[0339] In some possible implementations, one or more computing devices in this communication-aware integrated cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 15 illustrates one possible implementation. As shown in Figure 15, two computing devices 1300A and 1300B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device.

[0340] In this type of possible implementation, the memory 1306 in computing device 1300A stores a processing module, which is used to execute instructions for the functions of processing device 1201 in the embodiment shown in FIG. 12. Meanwhile, the memory 1306 in computing device 1300B stores a sensing module, which is used to execute instructions for the functions of sensing device 1202 in the embodiment shown in FIG. 12.

[0341] It should be understood that the functions of computing device 1300A shown in Figure 15 can also be performed by multiple computing devices 1300. Similarly, the functions of computing device 1300B can also be performed by multiple computing devices 1300.

[0342] This application also provides another communication-sensing integrated cluster. The connection relationship between the computing devices in this communication-sensing integrated cluster can be similar to the connection method of the communication-sensing integrated cluster shown in FIG15. The difference is that the memory 1306 in one or more computing devices 1300 in this communication-sensing integrated cluster can store the same instructions for executing the methods provided in any of the above embodiments.

[0343] In some possible implementations, the memory 1306 of one or more computing devices 1300 in the integrated communication and sensing cluster may also store partial instructions for executing the methods provided in any of the above embodiments. In other words, a combination of one or more computing devices 1300 can jointly execute instructions for executing the methods provided in any of the above embodiments.

[0344] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform the methods provided in any of the above embodiments.

[0345] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device to perform the method provided in any of the above embodiments.

[0346] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0347] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A system integrating communication and sensing, characterized in that, The system includes a processing device and multiple sensing devices. For each sensing device, the sensing device includes a first optical interface, and the processing device includes a second optical interface. The first optical interface of the sensing device is connected to the second optical interface of the processing device through a point-to-multipoint optical connection. The sensing device is used to acquire first sensing data; The sensing device is also used to send the first sensing data to the processing device via the optical connection; The processing device is used to perform target processing on the first sensing data to obtain the second sensing data.

2. The system of claim 1, wherein, The first sensing data includes background data and foreground data. The background data is the same as the third sensing data and the first sensing data. The foreground data is the different data between the third sensing data and the first sensing data. The third sensing data is the previous frame data acquired by the sensing device.

3. The system as described in claim 2, characterized in that, The sensing device is configured to compress the foreground data based on a first compression ratio to obtain first compressed data, and to compress the background data based on a second compression ratio to obtain second compressed data, wherein the first compression ratio is less than the second compression ratio; The first compressed data and the second compressed data are sent to the processing device via the optical connection; The processing device is further configured to decompress the first compressed data to obtain the foreground data, and to decompress the second compressed data to obtain the background data; and to obtain the first perceived data based on the foreground data and the background data.

4. The system as described in claim 2, characterized in that, The processing device includes the background data; The sensing device is configured to compress the foreground data based on a first compression ratio to obtain first compressed data; and to send the first compressed data to the processing device via the optical connection. The processing device is further configured to decompress the first compressed data to obtain the foreground data; and to obtain the first perceived data based on the foreground data and the background data.

5. The system as described in claim 3 or 4, characterized in that, The first compression ratio is determined based on the bandwidth that the sensing device can actually use.

6. The system according to any one of claims 2-5, characterized in that, The processing device includes fourth sensing data, which is data received by the processing device in the previous step. At least one frame of sensing data is spaced between the first sensing data and the fourth sensing data. The processing device is further configured to: Based on the changing pattern of the foreground data in the fourth perception data, the first perception data and the fourth perception data, at least one frame of fifth perception data located between the fourth perception data and the first perception data is obtained; The target processing is performed on the at least one frame of fifth-sensory data.

7. The system of any of claims 2-6, wherein, The processing device is used for: Allocate computing resources based on the amount of data in the foreground data; Based on the allocated computing resources, the first sensing data is processed to obtain the second sensing data.

8. The system of any one of claims 1-7, wherein, The processing device is also used for: Based on the bandwidth required by each sensing device and the total bandwidth of the second optical interface, bandwidth is allocated to each sensing device.

9. The system according to any one of claims 1-8, characterized in that, The target processing includes one or more of the following: enhancement processing, noise reduction processing, target detection processing, target recognition processing, and perceptual data fusion or perceptual data stitching.

10. The system of any one of claims 1-9, wherein, The sensing device is a camera device, and the first sensing data is image data; or... The sensing device is a radar device, and the first sensing data is point cloud data; or... The sensing device is an audio acquisition device, and the first sensing data is audio data.

11. The system according to any one of claims 1-10, characterized in that, The system is either a passive optical network (PON) or a fiber-to-the-room (FTTR) network.

12. A sensing device, characterized by The sensing device includes a sensing module, a conversion module, and an optical interface; The sensing module is used to acquire sensing data; The conversion module is used to convert the sensed data into a target signal in a target communication protocol defined format, and to convert the target signal into an optical signal. The target communication protocol is the communication protocol adopted by the sensed device. The optical interface is used to transmit the optical signal.

13. The sensing device as described in claim 12, characterized in that, The target communication protocol includes one or more of the following: Passive Optical Network (PON) protocol, Fiber to the Room (FTTR) protocol, or Ethernet protocol.

14. The sensing device as described in claim 12 or 13, characterized in that, The sensing device also includes a compression module. The compression module is used to perform shallow compression on the sensed data to obtain compressed data; The conversion module is used to convert the compressed data into the target signal, and the target signal into the optical signal.

15. The sensing device according to any one of claims 12-14, characterized in that, The sensing device also includes a wireless communication module, which is used to generate a wireless local area network.

16. The sensing device according to any one of claims 12-15, characterized in that, The sensing device also includes a power module, which is used to provide power to the sensing module and the conversion module.

17. The sensing device according to any one of claims 12-15, characterized in that, The optical interface is a photoelectric composite interface, and it is also used to provide power to the sensing module and the conversion module.

18. The sensing device according to any one of claims 12-17, characterized in that, The sensing device is a camera device, and the sensing data is image data; or... The sensing device is a radar device, and the sensing data is point cloud data; or... The sensing device is an audio acquisition device, and the sensing data is audio data.

19. A processing device, comprising: The processing device is a device included in a communication and sensing integrated system. The system also includes multiple sensing devices, each of which includes a first optical interface. The processing device includes a second optical interface, a conversion module, and a processing unit. The second optical interface is connected to the first optical interface of each sensing device through a point-to-multipoint optical connection. The second optical interface is used to receive at least one optical signal through the optical connection. The at least one optical signal is obtained by at least one sensing device acquiring at least one first sensing data and converting the at least one first sensing data. The plurality of sensing devices includes the at least one sensing device. The conversion module is used to convert the at least one optical signal into the at least one first sensing data; The processing unit is used to perform target processing on the at least one first sensing data to obtain at least one second sensing data.

20. The processing apparatus as described in claim 19, characterized in that, The processing equipment also includes: a scheduling unit and a computing resource pool; For each piece of first-sensory data, the scheduling unit is used to allocate computing resources from the computing resource pool to the first-sensory data based on the amount of data in the first-sensory data. The processing unit is used to perform the target processing on the first perceived data based on the allocated computing resources.

21. The processing device according to claim 19 or 20, characterized by The processing unit is further configured to allocate bandwidth to each sensing device based on the bandwidth required by each sensing device and the total bandwidth of the second optical interface.

22. The processing device of any of claims 19-21, wherein, The at least one optical signal is obtained by converting at least one compressed data, wherein the at least one compressed data is obtained by shallowly compressing the at least one first sensing data; The conversion module is used to convert the at least one optical signal into the at least one compressed data; The processing unit is further configured to decompress the at least one compressed data to obtain the at least one first sensing data.

23. The processing device of any of claims 19-22, wherein, The target processing includes one or more of the following: enhancement processing, noise reduction processing, target detection processing, target recognition processing, and perceptual data fusion or perceptual data stitching.

24. The processing apparatus according to any one of claims 19-23, characterized in that, The second optical interface is a photoelectric composite interface, and it is also used to provide power to the plurality of sensing devices.

25. The processing apparatus according to any one of claims 19-24, characterized in that, The system is either a passive optical network (PON) or a fiber-to-the-room (FTTR) network.

26. A method of communication, comprising: The communication method is applied to a communication and sensing integrated system. The system includes a processing device and multiple sensing devices. For each sensing device, the sensing device includes a first optical interface, and the processing device includes a second optical interface. The first optical interface of the sensing device is connected to the second optical interface of the processing device through a point-to-multipoint optical connection. The sensing device acquires first sensing data; The sensing device sends the first sensing data to the processing device via the optical connection; The processing device performs target processing on the first sensing data to obtain the second sensing data.

27. The method as described in claim 26, characterized in that, The first sensing data includes background data and foreground data. The background data is the same as the third sensing data and the first sensing data. The foreground data is the different data between the third sensing data and the first sensing data. The third sensing data is the previous frame data acquired by the sensing device.

28. The method of claim 27, wherein, The sensing device sends the first sensing data to the processing device via the optical connection, including: The sensing device compresses the foreground data based on a first compression ratio to obtain first compressed data, and compresses the background data based on a second compression ratio to obtain second compressed data, wherein the first compression ratio is less than the second compression ratio; The sensing device sends the first compressed data and the second compressed data to the processing device through the optical connection; Before the processing device performs target processing on the first sensed data, it further includes: The processing device decompresses the first compressed data to obtain the foreground data, and decompresses the second compressed data to obtain the background data; The processing device obtains the first perceived data based on the foreground data and the background data.

29. The method of claim 27, wherein, The processing device includes the background data; The sensing device sends the first sensing data to the processing device via the optical connection, including: The sensing device compresses the foreground data based on a first compression ratio to obtain first compressed data; The sensing device sends the first compressed data to the processing device via the optical connection; Before the processing device performs target processing on the first sensed data, it further includes: The processing device decompresses the first compressed data to obtain the foreground data; The processing device obtains the first perceived data based on the foreground data and the background data.

30. The method of claim 28 or 29, wherein, The first compression ratio is determined based on the bandwidth that the sensing device can actually use.

31. The method according to any one of claims 27-30, characterized in that, The processing device includes fourth sensing data, which is data received by the processing device in a previous instance, and the first sensing data and the fourth sensing data are spaced at least one frame of sensing data apart. The method further includes: The processing device acquires at least one frame of fifth sensing data located between the fourth sensing data and the first sensing data based on the changing pattern of the foreground data in the fourth sensing data, the first sensing data, and the fourth sensing data. The processing device performs the target processing on the at least one frame of fifth sensing data.

32. The method according to any one of claims 27-31, characterized in that, The processing device performs target processing on the first sensed data to obtain second sensed data, including: The processing device allocates computing resources based on the amount of data in the foreground data. The processing device performs target processing on the first sensing data based on the allocated computing resources to obtain the second sensing data.

33. The method according to any one of claims 26-32, characterized in that, The method further includes: The processing device allocates bandwidth to each sensing device based on the bandwidth required by each sensing device and the total bandwidth of the second optical interface.

34. The method of any one of claims 26-33, wherein, The target processing includes one or more of the following: enhancement processing, noise reduction processing, target detection processing, target recognition processing, and perceptual data fusion or perceptual data stitching.

35. The method according to any one of claims 26-34, characterized in that, The sensing device is a camera device, and the first sensing data is image data; or... The sensing device is a radar device, and the first sensing data is point cloud data; or... The sensing device is an audio acquisition device, and the first sensing data is audio data.

36. The method according to any one of claims 26-35, characterized in that, The system is either a passive optical network (PON) or a fiber-to-the-room (FTTR) network.

37. A cluster of computing devices, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 26-36.

38. A computer-readable storage medium, characterized in that, Includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 26-36.

39. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster performs the method as described in any one of claims 26-36.

Images