Apparatus, method, medium and computer program product for collaborative awareness

Abstract

The invention relates to a device, a method, a medium and a computer program product for collaborative awareness. An electronic device for a first device side comprises: a processing circuit configured to: receive model information of a collaborative perception model of a second device; determining the difference between the collaborative perception model of the second equipment and the collaborative perception model of the first equipment according to the model information of the collaborative perception model of the second equipment; and according to the difference, determining a collaborative perception strategy for the first device and the second device.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to devices and methods for collaborative sensing. Background Technology

[0002] Collaborative sensing is a method that utilizes network-connected devices and sensors to jointly perceive the surrounding environment. By connecting multiple devices and sensors together, they can share and integrate data, and improve the accuracy and efficiency of perception and identification through collaborative work. Collaborative sensing technology can be used in fields such as smart cities, intelligent transportation, and environmental monitoring to achieve more efficient and reliable environmental perception and data processing.

[0003] For example, cooperative perception can be applied to autonomous driving (AV) in vehicles to improve or solve problems that may be encountered in single-vehicle perception, such as environmental occlusion, long-range perception limitations, and sensor noise. Depending on the type of agent, cooperative perception can be divided into vehicle-to-vehicle cooperative perception and vehicle-to-infrastructure cooperative perception, etc. Summary of the Invention

[0004] A brief overview of this disclosure is given below to provide a basic understanding of some aspects of it. However, it should be understood that this overview is not an exhaustive summary of this disclosure. It is not intended to identify key or essential parts of this disclosure, nor is it intended to limit the scope of this disclosure. Its purpose is merely to present certain concepts of this disclosure in a simplified form as a prelude to the more detailed description that follows.

[0005] In practical deployments, vehicles and infrastructure manufactured by different companies may be equipped with different sensors and employ different perception models and fusion algorithms. Some collaborative perception models known to the inventors of this application cannot determine whether these models with different parameters and / or structures can be fused, or what the fusion effect would be. Furthermore, some collaborative perception models do not comprehensively consider other factors that may affect perception fusion, such as system bandwidth, transmission power, computing power, and latency.

[0006] In view of one or more of the above problems, this disclosure provides a mechanism for collaborative sensing that can make strategic decisions for collaborative sensing based on the differences between different models, enabling multi-node collaborative sensing between heterogeneous models, thereby improving the collaborative sensing effect.

[0007] According to one aspect of this disclosure, an electronic device for a first device side is provided. The electronic device may include processing circuitry configured to: receive model information of a collaborative sensing model of a second device; determine, based on the model information of the collaborative sensing model of the second device, a difference between the collaborative sensing model of the second device and the collaborative sensing model of the first device; and determine, based on the difference, a collaborative sensing strategy for the first device and the second device.

[0008] According to another aspect of this disclosure, an electronic device for a second device side is provided. The electronic device may include: a processing circuit configured to: provide a first device with model information of a collaborative sensing model of the second device, so that the first device, based on the model information of the collaborative sensing model of the second device, determines a difference between the collaborative sensing model of the second device and the collaborative sensing model of the first device, and determines a collaborative sensing strategy for the first device and the second device based on the difference; and receive the collaborative sensing strategy from the first device.

[0009] According to another aspect of this disclosure, a collaborative sensing method for a first device is provided. The method may include: receiving model information of a collaborative sensing model of a second device; determining, based on the model information of the collaborative sensing model of the second device, a difference between the collaborative sensing model of the second device and the collaborative sensing model of the first device; and determining, based on the difference, a collaborative sensing strategy for the first device and the second device.

[0010] According to another aspect of this disclosure, a collaborative sensing method for a second device is provided. The method may include: providing a first device with model information of a collaborative sensing model of the second device, so that the first device can determine the difference between the collaborative sensing model of the second device and the collaborative sensing model of the first device based on the model information of the collaborative sensing model of the second device, and determine a collaborative sensing strategy for the first device and the second device based on the difference; and receiving the collaborative sensing strategy from the first device.

[0011] According to another aspect of this disclosure, a computer-readable storage medium is provided, including executable instructions that, when executed by an information processing apparatus, cause the information processing apparatus to perform a collaborative sensing method according to this disclosure.

[0012] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when run by a processor, causes the processor to perform a collaborative sensing method according to this disclosure. Attached Figure Description

[0013] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the specification, serve to explain the principles of this disclosure.

[0014] This disclosure will be more clearly understood with reference to the accompanying drawings and the following detailed description, wherein:

[0015] Figure 1 This is a schematic diagram illustrating a collaborative sensing scenario;

[0016] Figure 2 This is an exemplary configuration block diagram illustrating an electronic device for a first device side according to an embodiment of the present disclosure;

[0017] Figure 3 This is an exemplary flowchart illustrating a collaborative sensing method for a first device side according to an embodiment of the present disclosure;

[0018] Figure 4 This is an exemplary configuration block diagram illustrating an electronic device for a second device side according to an embodiment of the present disclosure;

[0019] Figure 5 This is an exemplary flowchart illustrating a collaborative sensing method for a second device side according to an embodiment of the present disclosure;

[0020] Figure 6 This is an exemplary sequence diagram illustrating a collaborative sensing method according to embodiments of the present disclosure;

[0021] Figure 7 This is a block diagram illustrating a first example of an illustrative configuration of an RSU according to an embodiment of the present disclosure;

[0022] Figure 8 This is a block diagram illustrating a second example of an illustrative configuration of an RSU according to an embodiment of the present disclosure;

[0023] Figure 9 This is a block diagram illustrating an example of a schematic configuration of a smartphone according to an embodiment of the present disclosure; and

[0024] Figure 10 This is a block diagram illustrating an example of a schematic configuration of a car navigation device according to an embodiment of the present disclosure. Detailed Implementation

[0025] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0026] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0027] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0028] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0029] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0030] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0031] Figure 1 An exemplary scenario 100 of collaborative perception is shown.

[0032] Scenario 100 illustrates a road on which vehicles travel, which may include multiple vehicles 110 (in Figure 1 (Illustrated as 110a and 110b). Vehicle 110 can be any vehicle with cooperative perception capabilities, including but not limited to cars, trucks, buses, motorcycles, school buses, police cars, etc. Additionally, the vehicle can be a gasoline vehicle, an electric vehicle, or a hybrid vehicle.

[0033] Each vehicle 110 may be equipped with one or more sensors for sensing information related to vehicle operation, information related to the surrounding environment, etc. Sensors may include, but are not limited to, radar (e.g., ultrasonic radar, millimeter-wave radar, etc.), lidar (LiDAR), cameras (e.g., still cameras, stereo cameras, high-definition cameras, etc.), motion sensors (e.g., inertial measurement units, IMUs, etc.), and position sensors (e.g., GPS receivers, etc.). Furthermore, those skilled in the art may, according to actual needs, install other types of sensors to sense the required information.

[0034] Each vehicle 110 can utilize perception fusion (e.g., a neural network-based multimodal fusion perception algorithm) to fuse sensor data from different modalities to obtain, for example, a bird's-eye view (BEV) for subsequent processing. Depending on the stage of the data fusion, perception fusion can be categorized into early fusion, feature-level fusion (intermediate fusion), and late fusion. Early fusion directly fuses the raw sensor data, feature-level fusion fuses data in the feature space, and late fusion fuses the prediction (decision) results from each modality.

[0035] For example, each vehicle 110 may be provided with a computing unit, enabling the vehicle 110 to have the computing power (computing power) required to implement the vehicle 110's perception system (e.g., perception fusion and collaborative perception described below). This computing unit may include, but is not limited to, circuits such as integrated circuits (ICs), application-specific integrated circuits (ASICs), programmable hardware devices such as field-programmable gate arrays (FPGAs), central processing units (CPUs), graphics processing units (GPUs), tensor processing units (TPUs), neural processing units (NPUs), and / or combinations thereof.

[0036] While each vehicle 110 can achieve single-vehicle perception by fusing data from its various sensors, relying solely on single-vehicle perception can present several challenges. For example, sensors may have range limitations, preventing single-vehicle perception from detecting distant situations. Furthermore, environmental occlusion in real-world deployment scenarios can create blind spots in single-vehicle perception. Additionally, sensor noise can degrade perception performance, and in some cases, sensor failure / damage can significantly impair perception capabilities.

[0037] Due to the performance limitations of single-vehicle perception, some or all of the components of vehicle 110 can perform collaborative perception with other vehicles or other equipment (such as roadside equipment) to improve perception performance (e.g., expand the field of view) and thus assist driving. In collaborative perception, information from various collaborative perception participants (e.g., vehicle 110) (e.g., sensor data, intermediate features, prediction / decision results, model data, etc.) can be shared and fused. Similar to perception fusion, collaborative perception methods (fusion methods) can also include early fusion, feature-level fusion (intermediate fusion), late fusion, etc. Vehicle 110 can utilize neural networks (e.g., CNN, Transformer, etc.) as collaborative perception models for collaborative perception.

[0038] Based on their roles in collaborative sensing, vehicles 110 can be divided into primary vehicle 110a and collaborating vehicle 110b. Primary vehicle 110a is the initiator of collaborative sensing and has the authority to manage and configure collaborative sensing. Collaborating vehicle 110b is the responder of collaborative sensing and sends various information required for collaborative sensing (e.g., sensor data, model parameters, etc.) to primary vehicle 110a.

[0039] As an example, not a limitation, Figure 1 The illustration shows an exemplary scenario of collaborative perception between a master vehicle 110a and three cooperative vehicles 110b. Figure 1As shown, for example, the main vehicle 110a is entering an intersection and may be dissatisfied with its current perception performance (e.g., there are blind spots). To perceive the blind spots and distant situations, the main vehicle 110a initiates cooperative perception to request assistance from other nearby vehicles (e.g., cooperative vehicle 110b) to complete environmental perception of the area of ​​interest. In response to this cooperative perception request, N (e.g., N=3) cooperative vehicles 110b assist in completing environmental perception of the area of ​​interest, for example, by sending environmental data (e.g., BEV) of this area of ​​interest to the main vehicle 110a. The main vehicle 110a uses its own perceived environmental data and the environmental data from the cooperative vehicles 110b to perform perception fusion, thereby, for example, expanding its field of view, eliminating blind spots, and improving perception performance.

[0040] For example, vehicle 110 may be equipped with a communication unit, such as an on-board unit (OBU) based on a vehicle-to-everything (V2X) communication protocol and / or a mobile intelligent terminal carrying the V2X communication protocol. The main vehicle 110a and the cooperating vehicle 110b can exchange messages based on a V2X communication protocol (e.g., C-V2X (Cellular-Vehicle to everything)).

[0041] It should be understood that the above-mentioned message interaction based on the vehicle network communication protocol is only an example of the message interaction between vehicles 110 disclosed herein. Those skilled in the art can also use other communication methods according to actual needs, as long as the message interaction between the above-mentioned devices can be realized.

[0042] It should be noted that, although Figure 1 The example of vehicle-to-vehicle cooperation illustrates the cooperative perception between the master vehicle 110a and the cooperating vehicle 110b. However, the cooperative perception scenarios applicable to this disclosure can also include vehicle-to-infrastructure cooperative perception between vehicles and roadside infrastructure. For example, in some embodiments, Figure 1 Scenario 100 may also include one or more Road Side Units (RSUs) 120, which may also participate in the aforementioned collaborative sensing. RSUs 120 may be, for example, intelligent infrastructure used to sense road conditions, traffic conditions, and other information, and to exchange this information with vehicles or other RSUs via a communication network. Examples of RSUs 120 include, but are not limited to, smart poles and smart base stations.

[0043] Similar to vehicle 110, RSU 120 may also possess certain sensing, computing, and communication capabilities. For example, RSU 120 may be equipped with one or more sensors to sense information related to vehicle movement and the surrounding environment. RSU 120 may also include a computing unit to provide the computing power required to implement the sensing system. For instance, RSU 120 may have higher computing power than vehicle 110. Furthermore, RSU 120 can interact with vehicle 110 and / or other RSUs 120 via vehicle-to-everything (V2X) communication protocols.

[0044] As mentioned above, RSU 120 can also participate. Figure 1 The cooperative sensing is illustrated in the diagram. For example, RSU 120 can act as either the initiator or the responder of cooperative sensing. For instance, RSU 120 can initiate cooperative sensing and request one or more cooperating vehicles 110b to assist in completing environmental sensing of the area of ​​interest. Alternatively, RSU 120 can respond to cooperative sensing requests from the master vehicle 110a or other RSU 120 and send its sensing data to the initiator of the cooperative sensing.

[0045] It should be understood that the devices for collaborative sensing described in this disclosure are not limited to vehicles and RSUs, but can also be other devices with the need and capability for collaborative sensing, such as portable UEs, smartphones, wearable devices, drones, cameras and surveillance equipment, sensor networks, etc. The following description will use vehicles and RSUs as examples.

[0046] Different vehicles and RSUs may use different cooperative perception models for cooperative perception. These models can be implemented using neural network models (e.g., distributed neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), and long short-term memory networks (LSTMs). For example, due to differences in manufacturers, requirements, etc., the main vehicle 110a, the cooperative vehicle 110b, and the RSU 120 may use different cooperative perception models. These models may differ in their model structure and parameters (e.g., the model structure and parameters of the neural network model used), perception capabilities (e.g., depth perception capability, color perception capability, perception accuracy, etc.). Therefore, after receiving information from the collaborating party (e.g., cooperative vehicle 110b), the initiator of cooperative perception (e.g., the main vehicle 110a) needs to determine whether the information from the cooperative vehicle 110b can meet the cooperative perception requirements of the main vehicle 110a, or whether the perception data from the cooperative vehicle 110b can be fused with the data from the main vehicle 110a.

[0047] In this disclosure, a collaborative perception strategy decision is made considering the differences between different models (e.g., differences in model structure and parameters, differences in perception capabilities, etc.). Next, referring to... Figures 2 to 6 This section provides a detailed description of the collaborative sensing scheme disclosed herein.

[0048] Figure 2 An exemplary configuration block diagram of an electronic device 200 for a first device side according to an embodiment of the present disclosure is shown. In the context of this disclosure, the first device is, for example, an initiator of collaborative sensing, or has the authority to manage and configure collaborative sensing. For example, the first device may correspond to... Figure 1 The main vehicle shown is 110a or RSU 120.

[0049] In some embodiments, the electronic device 200 may include processing circuitry 210. The processing circuitry 210 of the electronic device 200 provides various functions of the electronic device 200. In some embodiments, the processing circuitry 210 of the electronic device 200 may be configured to perform a cooperative sensing method for a first device side.

[0050] Processing circuitry 210 can refer to various implementations of digital circuitry, analog circuitry, or mixed-signal (analog and digital combination) circuitry that perform functions in a computing system. Processing circuitry can include, for example, circuitry such as integrated circuits (ICs), application-specific integrated circuits (ASICs), portions or circuitry of a single processor core, an entire processor core, a single processor, programmable hardware devices such as field-programmable gate arrays (FPGAs), and / or systems comprising multiple processors.

[0051] In some embodiments, the processing circuit 210 may include a model information acquisition unit 220, a model difference determination unit 230, and a strategy determination unit 240, which are respectively configured to perform the following description. Figure 3 Steps S310 to 330 of the cooperative sensing method 300 for the first device side shown.

[0052] In some embodiments, the electronic device 200 may further include a memory (not shown). The memory of the electronic device 200 may store information generated by the processing circuitry 210, as well as programs and data for the operation of the electronic device 210. The memory may be volatile memory and / or non-volatile memory. For example, the memory may include, but is not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), and flash memory. Furthermore, the electronic device 200 may be implemented at the chip level, or it may be implemented at the device level by including other external components.

[0053] Figure 3An exemplary flowchart of a cooperative sensing method 300 for a first device side according to an embodiment of the present disclosure is shown. This cooperative sensing method can be used, for example, for... Figure 2 The electronic device 200 shown.

[0054] like Figure 3 As shown, in step S310, the model information acquisition unit 220 receives model information of the collaborative sensing model of the second device. In the context of this disclosure, the second device is, for example, a responder or collaborator in the collaborative sensing process. For example, the second device may correspond to... Figure 1 The collaborating vehicles are 110b or RSU 120.

[0055] The model information of the collaborative sensing model can include any model information that can participate in collaborative sensing. Table 1 below shows an exemplary model information table, including the model's structure and parameters, the model's differential index parameters, and the model's intermediate features on a specific dataset. The model information of the collaborative sensing model received in step S310 may include one or more items from Table 1 below. It should be understood that the above model information is merely an example, and the model information according to embodiments of this disclosure may also include other information about the collaborative sensing model.

[0056] Table 1: Model Information Table

[0057]

[0058] In some embodiments, the structure and parameters of the model may include the structure (e.g., the structure of neural networks such as ResNet and FasterNet, and the data size of the layers (e.g., 128×128, 64×64, etc.)) and parameter (e.g., weights) information of the base layer and / or intermediate fusion layer of the collaborative perception model. The term "base layer" may refer to the part (module) in the collaborative perception model that first processes the raw input data (e.g., RGB image) and extracts features. The term "intermediate fusion layer" may refer to the part (module) in the collaborative perception model that fuses information from various vehicles and / or RSUs (e.g., BEV). By obtaining the structure and parameter information of the base layer and / or intermediate fusion layer of the collaborative perception model of the second device, the first device can, for example, determine whether the collaborative perception model of the second device can meet the collaborative perception requirements of the first device, and determine a collaborative perception strategy accordingly.

[0059] In some embodiments, the structure and parameters of the model may include modal information of the source data processed by the base layer of the collaborative sensing model. Depending on the source of the source data (e.g., sensing data from radar, lidar, cameras, etc.), the source data may have different modalities. The modality of the source data can indicate the sensing capability of the sensing source; different modalities may have different advantages and disadvantages in sensing capability. For example, lidar can accurately acquire depth and shape but cannot sense color, while a camera can sense color but may not have depth and distance sensing capabilities. By acquiring the modal information of the source data processed by the base layer of the collaborative sensing model of the second device, the first device can, for example, select modal information with its own deficiencies from the second device to assist in collaborative sensing, thereby improving the effect of sensing fusion.

[0060] In some embodiments, the structure and parameters of the model may include information about the training dataset used to train the collaborative perception model (e.g., the origin, type, name, etc. of the training dataset). Information about the training dataset can characterize the ability to perceive the environment and may affect the effectiveness of perception fusion. For example, the fusion effect of two models trained using datasets of roads from China and the United States, respectively, may be poor because road signs differ between the two countries. Therefore, the first device can determine a collaborative perception strategy based on information from the training dataset of the second device, such as selecting a collaborative perception model from the second device that has a training dataset that matches the training dataset of the first device, to assist in collaborative perception.

[0061] The above description describes the model information of the collaborative sensing model of the second device received by the model information acquisition unit 220 of the first device. This model information can be the structure and parameters of the collaborative sensing model, which the first device uses to determine the differences between the collaborative sensing models of the second device and the first device. In some embodiments, the first device may also receive quantified difference information from the second device, i.e., the difference index parameters of the collaborative sensing model of the second device.

[0062] More specifically, the model's differentiated performance parameters can be performance dimensions of the model in perception tasks, and can include, but are not limited to, quantified values ​​of capabilities such as depth perception, speed perception, color perception, shape perception, size perception, object category recognition, perception range, and perception accuracy (e.g., centimeter or millimeter level). For example, quantifying depth perception capability can be expressed as the error in a vehicle's perception of the distance to objects within a specific range (e.g., 100 meters) not exceeding a specific value (e.g., ±20 centimeters). Similarly, quantifying object category recognition capability can be expressed as a list of object types and their confidence levels. Other performance dimensions can be similarly quantified.

[0063] As mentioned above, models processing source data of different modalities may have different differential metrics. For example, a LiDAR might be strong in depth perception but unable to perceive color, while a camera might be weak in depth perception but capable of color perception. Therefore, the differential metric parameters of a model can reveal its perception capabilities across different performance dimensions. By acquiring the differential metric parameters of the collaborative perception model of the second device, the first device can intuitively understand its perception capabilities, thus determining the differences between its collaborative perception model and its own.

[0064] In some embodiments, the model information of the collaborative sensing model of the second device received by the model information acquisition unit 220 of the first device may further include intermediate features of the model on a specific dataset. For example, it may include intermediate feature values ​​generated by the collaborative sensing model for a specific dataset. More specifically, the intermediate feature values ​​may be corresponding intermediate fusionable features (e.g., BEV or features extracted from BEV) obtained by inputting the specific dataset into the collaborative sensing model and performing only the calculation from the initial base layer to the intermediate fusion layer (without entering the complete task head). The intermediate fusionable features may also be used to determine the differences between the collaborative sensing model of the first device and the model of the first device.

[0065] In some embodiments, the specific dataset mentioned above may include a benchmark dataset sent by the first device to the second device. In such embodiments, the second device may indicate to the first device a request for benchmark dataset matching. Alternatively, the first device may determine on its own whether benchmark dataset matching is needed (e.g., the first device believes that the received model information is insufficient to determine the differences between models) and send the benchmark dataset to the second device. The benchmark dataset may be, for example, a dataset of a specific size (e.g., 20 frames) recently sampled by the first device, or it may be a dataset obtained by the first device from other sources (e.g., a general database). The first device may select and send the appropriate benchmark dataset based on the current environment (surrounding road categories, scene categories, etc.). Alternatively, the specific dataset may also include historical datasets processed by the second device's collaborative perception model. In other words, the second device may upload its own historical intermediate features as intermediate features of the model on the specific dataset.

[0066] Next, in step S320, the model difference determination unit 230 determines the difference between the collaborative sensing model of the second device and the collaborative sensing model of the first device based on the model information of the collaborative sensing model of the second device. The model difference determination unit 230 can determine the difference between the models based on one or more of the model structure and parameters, the model differentiation index parameters, and the intermediate features of the model on a specific dataset received in step S310.

[0067] For example, upon receiving the structure and parameters of the model of the second device, the model difference determination unit 230 can calculate the Euclidean distance, variance, covariance, etc. of the structure and parameters of the models of the first device and the second device to quantify the difference between the two models (e.g., to obtain the discriminative power of the parameters). The smaller the difference, the more suitable the two models may be for fusion.

[0068] Suppose the models of the first and second devices are A and B, respectively (assuming that models A and B are completely isomorphic), and their weights are θ. A and θ B Then, the Euclidean distance between models A and B can be calculated using the following formula.

[0069]

[0070] Where, θ A,i and θ B，i These represent the weight vector θ. A and θ B The i-th element, where n is the dimension of the weight vector.

[0071] Additionally, the variance of the weight vectors for each model can be calculated and compared. For example, the variance of the difference between the weight vectors of models A and B can be calculated using the following formula.

[0072]

[0073] Where μ is θ A -θ B The mean of can be expressed as:

[0074]

[0075] In addition, the covariance of the weight vectors of models A and B can be calculated using the following formula.

[0076]

[0077] in, and Represent θ A and θ B Mean:

[0078]

[0079]

[0080] Furthermore, the above example, using models A and B as complete isomorphism, illustrates how to calculate the Euclidean distance, variance, and covariance between models A and B. When models A and B have different structures (e.g., different neural network data sizes), the models can first be converted to isomorphic forms, and then the Euclidean distance, variance, and covariance between the two models can be calculated using the methods described above to quantify the differences between them.

[0081] In addition, the above methods for quantifying model differences (Euclidean, variance, covariance) are only examples. You can choose one or more of these methods to quantify model differences as needed, or you can choose other methods to quantify model differences.

[0082] In some embodiments, the model difference determination unit 230 can determine the differences between the modalities of the source data and the training datasets of the first device based on the modal information of the source data processed by the base layer of the collaborative perception model of the second device and the information of the training dataset, so as to provide a basis for subsequent screening of modalities and training datasets for collaborative perception fusion.

[0083] In some embodiments, upon receiving differential indicator parameters of the model from the second device, the model difference determination unit 230 can filter the differential indicator parameters based on the missing items of its own collaborative perception task. For example, the first device can determine that a certain capability of the second device can fill its own missing items, thereby determining that the second device may be a match for itself and suitable for perception fusion.

[0084] Furthermore, upon receiving intermediate features from a second device corresponding to a specific dataset (e.g., a benchmark dataset or a historical dataset), the model difference determination unit 230 can calculate the feature matching degree, which may include determining the corresponding fusion structure for the missing item dimension.

[0085] In step S330, the strategy determination unit 240 determines a collaborative sensing strategy for the first device and the second device based on the differences determined in step S320.

[0086] In some embodiments, multiple second devices may be present. For example, such as Figure 1 As shown, the main vehicle 110a can receive model information of the cooperative perception model from three cooperative vehicles 110b. At this time, the cooperative perception strategy can instruct one of the multiple second devices to perform cooperative perception with the first device. In other words, the cooperative perception strategy can indicate which second devices to perform cooperative perception fusion with.

[0087] Furthermore, in some embodiments, each second device may use multiple collaborative sensing models (which may have different algorithms, modalities, etc.) and may switch between these collaborative sensing models. In this case, the collaborative sensing strategy may specify the collaborative sensing model among the multiple collaborative sensing models used for collaborative sensing with the first device. Additionally, in some embodiments, the collaborative sensing strategy may also specify the frequency of fusion, etc., at which the devices participating in collaborative sensing perform data interaction for collaborative sensing according to this frequency.

[0088] According to the collaborative sensing method 300 disclosed herein, strategic decisions for collaborative sensing can be made based on the differences between different models, enabling multi-node collaborative sensing among heterogeneous models, thereby improving the collaborative sensing effect.

[0089] Referenced above Figure 2 and 3 A cooperative sensing device and method for a first device side, according to embodiments of this disclosure, have been described. Next, reference will be made to... Figure 4 and Figure 5 This describes a collaborative sensing device and method for a second device side according to embodiments of the present disclosure.

[0090] Figure 4 An exemplary configuration block diagram of an electronic device 400 for a second device side according to an embodiment of the present disclosure is shown. As described above, the second device may, for example, correspond to... Figure 1 The collaborating vehicles are 110b or RSU 120.

[0091] In some embodiments, the electronic device 400 may include processing circuitry 410. The processing circuitry 410 of the electronic device 400 provides various functions of the electronic device 400. In some embodiments, the processing circuitry 410 of the electronic device 400 may be configured to perform a cooperative sensing method for a second device side.

[0092] Processing circuitry 410 can refer to various implementations of digital circuitry systems, analog circuitry systems, or mixed-signal (analog and digital combination) circuitry systems that perform functions in a computing system. Processing circuitry can include, for example, circuitry such as integrated circuits (ICs), application-specific integrated circuits (ASICs), portions or circuitry of a single processor core, an entire processor core, a single processor, programmable hardware devices such as field-programmable gate arrays (FPGAs), and / or systems comprising multiple processors.

[0093] In some embodiments, the processing circuit 410 may include a model information providing unit 420 and a strategy acquisition unit 430, respectively configured to perform the following description. Figure 5 Steps S510 to 520 in the cooperative sensing method 500 for the second device side shown.

[0094] In some embodiments, the electronic device 400 may further include a memory (not shown). The memory of the electronic device 400 may store information generated by the processing circuitry 410, as well as programs and data for the operation of the electronic device 410. The memory may be volatile memory and / or non-volatile memory. For example, the memory may include, but is not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), and flash memory. Furthermore, the electronic device 400 may be implemented at the chip level, or it may be implemented at the device level by including other external components.

[0095] Figure 5 An exemplary flowchart of a collaborative sensing method 500 for a second device side according to an embodiment of the present disclosure is shown. This collaborative sensing method can be used, for example, for... Figure 4 The electronic device 400 shown.

[0096] like Figure 5 As shown, in step S510, the model information providing unit 420 provides the first device with the model information of the collaborative perception model of the second device, so that the first device can determine the difference between the collaborative perception model of the second device and the collaborative perception model of the first device based on the model information of the collaborative perception model of the second device, and determine the collaborative perception strategy for the first device and the second device based on the difference.

[0097] The model information of the collaborative sensing model of the second device may include, for example, references. Figure 3 The model information described in step S310 includes, but is not limited to, the model's structure and parameters, the model's differential index parameters, and one or more of the model's intermediate features on a specific dataset.

[0098] After receiving the model information from the collaborative sensing model of the second device, the first device can refer to... Figure 3 As described in step S320, the difference between the collaborative sensing model of the second device and the collaborative sensing model of the first device is determined based on the model information. Then, the first device can, as with reference to... Figure 3 As described in step S330, a collaborative sensing strategy for the first and second devices is determined based on the difference.

[0099] In step S520, the policy acquisition unit 430 receives a collaborative sensing policy from the first device. For example, in an embodiment where the second device includes multiple collaborative sensing models, the second device can deploy the collaborative sensing model indicated in the collaborative sensing policy for collaborative sensing with the first device. The second device can send sensing data corresponding to the fusion method and fusion mode indicated in the collaborative sensing policy to the first device. Furthermore, the second device can send corresponding sensing data at the frequency indicated by the collaborative sensing policy.

[0100] Figure 6 An exemplary sequence diagram 600 is shown for a collaborative sensing method according to embodiments of the present disclosure. Sequence diagram 600 illustrates message interaction between three nodes performing the collaborative sensing method according to embodiments of the present disclosure. For example, node 1 and node 2 may correspond to a second device, and may correspond to... Figure 1 The collaborating vehicle 110b or RSU 120. Node 3 can correspond to the first device and can correspond to... Figure 1 The main vehicle shown is 110a or RSU 120.

[0101] It should be noted that, although Figure 6 The diagram illustrates message interaction between three nodes, but the collaborative sensing method according to this disclosure can also be applied to any number of nodes, including two or more nodes. Furthermore, these nodes can be other devices besides vehicles and RSUs, such as portable UEs, as long as they possess certain sensing, computing, and communication capabilities.

[0102] In step S601, Node 1 and Node 2 send their respective collaborative sensing model information to Node 3. The collaborative sensing model information may include, for example, reference... Figure 3 The model information described in step S310.

[0103] In some embodiments, before step S601, node 3 may send a collaborative sensing service request to node 1 and node 2, and node 1 and node 2 may send the aforementioned model information to node 3 in response to the collaborative sensing service request.

[0104] Alternatively, nodes 1 and 2 can proactively provide the aforementioned model information to node 3 in step S601, based on a predetermined positional relationship with node 3. For example, nodes 1 / 2 can determine that they are geographically close to node 3 and send model information to node 3, or nodes 1 / 2 can determine, based on map data (e.g., high-definition maps), that there are differences in perception capabilities between them and the first device (e.g., their blind spots do not overlap) and send model information to node 3.

[0105] Next, in step S606, node 3 determines the differences between its own collaborative perception model and the collaborative perception models of nodes 1 and 2. For example, node 3 can refer to... Figure 3 As described in step S320, the difference between its own collaborative perception model and the collaborative perception models of nodes 1 and 2 is determined based on the model information received in step S601 and / or the intermediate features received in step S603 described later.

[0106] In step S607, node 3 determines a collaborative sensing strategy based on the differences between the collaborative sensing models determined in step S606. Node 3 can refer to... Figure 3 As described in step S330, a collaborative sensing strategy for node 3 with nodes 1 and 2 is determined based on the differences identified in step S606.

[0107] In step S608, node 3 sends the collaborative sensing strategy determined in step S607 to node 1 and node 2.

[0108] In step S609, nodes 1 and 2 may send corresponding sensing data to node 3 in response to the collaborative sensing strategy received in step S608.

[0109] In some embodiments, optionally, if the model information of the collaborative sensing model includes intermediate features of the model on a specific dataset, the sequence diagram 600 may also include steps S602 and S603 for transmitting the benchmark dataset.

[0110] More specifically, in step S602, node 3 can send benchmark datasets to nodes 1 and 2. Node 3 can send benchmark datasets in response to requests from nodes 1 and 2 for matching benchmark datasets, or node 3 can determine on its own whether to send benchmark datasets. Node 3 can select and send the corresponding benchmark datasets based on the current environment (surrounding road categories, scene categories, etc.). Node 3 can send benchmark datasets to other nodes to test their corresponding performance based on performance deficiencies indicated by a node (e.g., node 3 itself). For example, if area coverage is missing, a benchmark dataset focusing on omnidirectional perception can be sent; if object perception details are missing, data containing rich object surfaces can be sent; if generalization performance is missing, data on scene diversity can be sent; if modality is missing, data corresponding to the modality can be sent.

[0111] In step S603, nodes 1 and 2 can send intermediate features as model information to node 3. In some embodiments, nodes 1 and 2 can input the benchmark dataset sent by node 3 in step S602 into their respective collaborative perception models to obtain corresponding intermediate fusionable features, and provide these intermediate features to node 3. Alternatively, nodes 1 and 2 can provide their respective historical intermediate features (i.e., intermediate features generated for historical datasets) to node 3.

[0112] Note that steps S602 and S603 can be omitted, can be used as a substitute for step S601, or can be performed together with step S601.

[0113] In some embodiments, optionally, in step S604, nodes 1 and 2 may send a retraining request to node 3. For example, when node 1 or node 2 is dissatisfied with the currently deployed collaborative perception model (e.g., the model of node 1 or node 2 performs poorly in the current environment, node 1 or node 2 enters a new environment, is repeatedly rejected for collaborative perception by other nodes, or is periodically updated, etc.), node 1 or node 2 may indicate its need for model retraining to node 3, indicating the type of retraining required for its model (e.g., federated learning, transfer learning, fine-tuning), and the structure involved in the retraining (including but not limited to classifying the model structure into modules such as base layers, intermediate fusion layers, task layers, adapter layers, etc., and indicating which module needs to be retrained). Note that step S604 may be omitted, and in some cases, node 3 may determine on its own whether node 1 or node 2 needs retraining.

[0114] In some embodiments, node 3 can also determine a retraining strategy based on the retraining request received in step S604. For example, node 3 can select an appropriate dataset (selecting the corresponding dataset based on the missing items of the collaborative perception performance indicated by node 1 or 2) and a learning process (including the determination of the retraining category, the matching between federated learning nodes, the nodes to be executed for retraining, etc.) for the retraining process of node 1 / node 2. In addition, node 3 can send the retraining strategy to node 1 and node 2.

[0115] In some embodiments, optionally, in step S605, nodes 1 and 2 may send a Cooperative Perception Message (CPM) to node 3 to provide node 3 with information related to cooperative perception other than model information. In step S607, node 3 may further consider the received cooperative perception messages from nodes 1 and 2 to determine a cooperative perception strategy (which will be described in detail below).

[0116] It should be noted that, Figure 6 The order of the steps shown is not limited to the order illustrated in the diagram. Figure 6 One or more steps shown may be executed in different orders and / or concurrently.

[0117] The above-described sequence diagram 600 illustrates the cooperative perception method according to this disclosure, which matches an appropriate cooperative perception model based on the differences between the cooperative perception models of node 3 and node 1 / node 2. This method is a model performance matching method oriented towards cooperative perception tasks. Some cooperative perception tasks may require greater differences between models (e.g., perception tasks at the same viewpoint to obtain more object details), while other cooperative perception tasks may require smaller differences between models (e.g., perception fusion at different viewpoints, which may require the models of each node to be closer to have an anchor point for fusion). Therefore, a cooperative perception task matching degree can be determined; for example, when the cooperative perception task requires greater differences between models, the task matching degree of the cooperative perception models of other nodes that are more different from the cooperative perception model of node 3 is set higher, and vice versa.

[0118] In some embodiments, optimization algorithms can be used to solve for the cooperative sensing strategy. The determination of the cooperative sensing objective (optimization objective) and the cooperative sensing strategy in the optimization algorithm will be described in detail below.

[0119] Collaborative perception target (optimization target)

[0120] In some embodiments, the collaborative sensing target can be set to be related to the collaborative sensing matching degree C. By calculating the collaborative sensing matching degree between the collaborative sensing model of the second device and the collaborative sensing model of the first device, the collaborative sensing target is optimized, thereby determining an appropriate collaborative sensing strategy. For example, the second device with the highest collaborative sensing matching degree among multiple second devices can be determined by calculation, and this second device can be identified as the device to be collaboratively sensing with the first device as the collaborative sensing strategy. Alternatively, the collaborative sensing model with the highest collaborative sensing matching degree among multiple collaborative sensing models of the second device can be determined by calculation, and this collaborative sensing model of the second device can be identified as the model to be collaboratively sensing with the first device as the collaborative sensing strategy.

[0121] In other embodiments, in addition to considering the matching degree of cooperative sensing tasks, other factors such as communication channel bandwidth, transmit power, and system latency may be considered when setting cooperative sensing targets. These factors can be indicated, for example, by cooperative sensing messages (CPM) received from the second device.

[0122] For example, a first device can receive cooperative sensing messages (CPMs) from one or more second devices. These CPMs (beacon information) can indicate one or more of the following: the location (e.g., relative location), speed (e.g., relative speed), computing power, data transmission volume, communication channel conditions (e.g., wireless communication quality), the type of sensor carried, sensing range, and / or the amount of sensing data. This step can be implemented, for example, by step S605 in sequence diagram 600. In step S605, node 1 and node 2 can send cooperative sensing messages to node 3.

[0123] By comprehensively considering the cooperative sensing target and the cooperative sensing messages received from the second device, a cooperative sensing strategy can be determined, which can achieve a balance between sensing accuracy, sensing range and system latency.

[0124] In some embodiments, the cooperative sensing target of the first device can be set to be related to one or more of the following: cooperative sensing task matching degree C, communication channel bandwidth constraint B, transmit power constraint P, and system delay T. The communication channel bandwidth constraint B and transmit power constraint P may be affected by the location, speed, and sensing range of the devices participating in the cooperative sensing, while the system delay T may be affected by the computing power, the amount of data transmitted, and the communication quality of the devices participating in the cooperative sensing. The constraints such as the communication channel bandwidth constraint B, transmit power constraint P, and system delay T can be determined based on the cooperative sensing messages.

[0125] Therefore, by setting the collaborative sensing target of the first device to be related to one or more of the following: collaborative sensing task matching degree C, communication channel bandwidth constraint B, transmission power constraint P, and system latency T, it is possible to comprehensively consider various factors such as the size of the computing task, the bandwidth of the wireless channel, and the computing capabilities of the vehicle and roadside units for joint optimization, thereby achieving a balance between system latency and sensing utility. This approach has universality in scenarios with complex communication and computing resource conditions and is more robust in dealing with complex environments.

[0126] Collaborative perception strategy

[0127] The embodiments described above describe a collaborative perception strategy that can instruct a second device and / or collaborative perception model for collaborative perception with a first device. In some scenarios, it is also desirable to instruct the fusion method of the collaborative perception model (e.g., data-level fusion (also known as early fusion), feature-level fusion (also known as mid-term fusion), result-level fusion (also known as late-term fusion), etc.) and the fusion mode (perception mode) of the data modalities of the first and second devices performing collaborative perception. This is because redundant vehicle-to-vehicle (V2V) collaboration latency exists in all collaborative vehicles, fusion methods, and perception modes. Inappropriate selection of collaborative vehicles can lead to impractical intelligent decisions by the primary vehicle, such as trading excessive V2V collaboration latency for a low improvement in the primary vehicle's perception performance. Furthermore, a single fusion method and a single perception mode can also limit the improvement of the primary vehicle's utility.

[0128] Considering the above, in some embodiments, the cooperative sensing strategy may indicate one or more of the following: a second device among the plurality of second devices for cooperative sensing with the first device, a fusion method for the cooperative sensing models of the first and second devices, and a fusion mode for the data modes of the first and second devices for cooperative sensing. Thus, on the one hand, the advantages and disadvantages of sensors can be complemented to improve sensing accuracy and range; on the other hand, it can cope with complex weather conditions and vehicle sensor configuration conditions.

[0129] Optimization problem of collaborative sensing strategy

[0130] In some embodiments, the problem of determining a cooperative sensing strategy (e.g., jointly optimizing the selection of cooperative vehicles, fusion methods, and sensing modes) can be modeled as a nonlinear mixed-integer optimization problem, which can be to maximize system utility (i.e., the optimization objective) while satisfying constraints such as system bandwidth and maximum transmit power. System utility can balance system latency calculated based on received cooperative sensing messages (beacon information) with sensing utility. For example, a first device can calculate sensing utility (e.g., including sensing coverage area and sensing accuracy) based on factors such as vehicle speed and sensing range to quantify sensing performance. The first device can also calculate system (overall) latency (e.g., including transmission latency for sending data and computation latency required for fusion) based on factors such as sensing data volume, transmitted data volume, vehicle speed, and / or communication channel conditions to quantify timeliness. System utility can be defined as the difference between sensing utility and system latency (e.g., a weighted difference). Determining a cooperative sensing strategy can aim to maximize this system utility.

[0131] Next, as an example, we will describe in detail the modeling and solution of the above-mentioned collaborative perception optimization problem.

[0132] The mathematical model is established using the following steps:

[0133] A1. The vehicles (including the main vehicle 110a and the cooperative vehicle 110b) first collect data. Various sensors can be equipped on the vehicles; three sensors are used as examples here: cameras, lidar, and radar. The amount of data collected by the camera of vehicle i... σ represents the resolution of the image captured by the camera, and σ is the number of bits used to store each pixel, i.e., the pixel depth, which can be 8 bits, 10 bits, etc. The data size of the point cloud acquired by LiDAR and radar are respectively... Here, θ represents the angular resolution of the data collected by the lidar and radar sensors, respectively, and μ represents the number of bits per unit point cloud. The size of the raw data collected locally is [size missing]. m i,1 m i,2 m i,3 These represent the number of data frames for the camera mode, lidar mode, and radar mode, respectively.

[0134] A2. The primary vehicle (e.g., primary vehicle 110a) receives and fuses data from cooperative vehicles (e.g., cooperative vehicle 110b). Taking mid-term fusion as an example, a data processing model, a data transmission model, and a data fusion model are established for the fusion process. Due to the mid-term fusion approach, the sensor-collected perception data is processed locally for feature extraction before being transmitted to the RSU. The vehicle's local processing model is as follows: The computational load for vehicle i is... With computational delay representation β1, β2, and β3 are the feature extraction computational complexity conversion coefficients for the three modes, u i This represents the computing power of vehicle i, i.e., the amount of computation it can process per unit time. The amount of data transmitted by vehicle i after fusion processing. Where γ1, γ2, and γ3 are the conversion coefficients for the amount of data extracted from the three modes of features.

[0135] A3. The vehicle data transmission model is as follows: Considering the impact of mobility on the small-scale fading coefficient, the small-scale fading coefficient of communication between vehicle i and the host vehicle is... in Δ is the small-scale fading coefficient for communication between vehicle i and the master vehicle in the previous time slot. i Let be a random variable that follows a complex Gaussian distribution. Channel correlation of vehicle i in two consecutive time slots Where J0(·) is the zeroth-order Bessel function, and the maximum Doppler frequency is... in λ is the carrier frequency.i Let τ be the speed of vehicle i, and τ be the time difference between two consecutive time slots. This represents the channel gain for communication between vehicle i and the host vehicle. in The large-scale fading coefficient is used to account for path loss and shadow fading. The data transmission rate of vehicle i. b i The bandwidth of a channel (usually measured in Hz) refers to the frequency range that the channel can use to transmit data. i W0 represents the power of the transmitted signal (usually measured in watts (W) or milliwatts (mW)), and W0 represents the power of interference or noise present in the channel (usually measured in watts (W) or milliwatts (mW)). The transmission delay of data sent by the cooperating vehicle i is also considered.

[0136] A4. The master vehicle data fusion model is as follows: After receiving the feature data transmitted by the cooperating vehicles, the master vehicle uses a feature data fusion method to perform data fusion. The computational cost required for fusion is... α i α is an indicator for vehicles to participate in coordination. i =1 indicates that the vehicle participates in the coordination; otherwise, α i =0, η M The conversion coefficients represent the computational complexity of the fusion result data in the later stages of fusion, and the computational latency required for fusion. u0 represents the computing power of the RSU. The overall system latency.

[0137] A5. Establish the optimization objective, i.e., the main vehicle utility function. Define the system's perceived utility function. Where IAera is the region of interest of the main vehicle, AS[Area] is the area size corresponding to the region Area, and λ i Let be the vehicle i's speed, and k(F) be the weight coefficient corresponding to different fusion methods, representing the perception accuracy. The main vehicle utility function is established as follows: Where F is the fusion method, A is the cooperative vehicle selection matrix, M is the sensor mode selection matrix, B is the bandwidth constraint, P is the maximum transmit power constraint, and ω1 and ω2 are the weights of the two sub-items, used to represent the emphasis on different performance characteristics.

[0138] When system latency is disregarded, early fusion leads to high perception accuracy, and selecting more cooperative vehicles results in greater perception coverage. Therefore, the system tends to favor early collaboration among multiple vehicles for heterogeneous data fusion, but this incurs extremely high system latency. In this disclosure, the primary vehicle utility function is expressed as the difference between perception utility and total system latency. By incorporating system latency as part of the system utility to influence perception decisions, the perception performance and timeliness of collaborative perception methods can be quantified.

[0139] In some embodiments, the initial value of the cooperative vehicle selection matrix A can be the cooperative vehicle indicated by the cooperative perception strategy determined by the cooperative perception method 300 / 500 described above. This optimization problem is solved jointly by further considering the influence of other factors to find the optimal cooperative perception strategy. In other embodiments, a cooperative perception matching degree C can be added to the master vehicle utility function for joint optimization to find the optimal cooperative perception strategy.

[0140] In some embodiments, the optimization problem described above can be solved using deep reinforcement learning. For example, in some embodiments, a deep Q-network (DQN) can be used to determine the cooperative perception policy. For instance, in this DQN, a "state" can correspond to information in the received cooperative perception messages, such as the vehicle's position, speed, computing power, etc.; an "action" can include fusion method selection, fusion mode selection, cooperative vehicle selection, etc.; and a "reward" can be associated with system utility (objective function). Furthermore, in some embodiments, such as those where both the first and second devices are vehicles, federated learning can be used to accelerate the training of the DQN to expedite the process of determining the cooperative perception policy.

[0141] As an example, the solution process for this optimization problem is described in detail below.

[0142] B1. Algorithm 2 below illustrates the collaborative training algorithm flow based on federated learning. To improve the convergence rate of the DQN model, federated learning technology is used to accelerate intelligent decision-making. Each vehicle trains its own DQN model locally. After a certain number of training rounds, it sends its DQN model to the RSU. The RSU receives and aggregates the DQN models sent to the collaborating vehicles, then distributes them to the collaborating vehicles. Upon receiving the DQN model from the RSU, the collaborating vehicles continue their local training. This process is repeated for a certain number of rounds until the collaborative training ends.

[0143]

[0144] B2. Algorithm 1, called in Algorithm 2 above, is a vehicle-to-vehicle cooperative perception algorithm based on DQN, as shown below. To find the optimal intelligent decision that maximizes the utility of the master vehicle, DQN is used to provide decisions on the fusion method selection F, the cooperative vehicle selection matrix A, and the sensor mode selection M.

[0145]

[0146] B3. A time-dimensional Markov Decision Process (MDP) model is used to model the optimization problem of maximizing the utility of the main vehicle. The state is defined as follows: at time slot t, environmental information including the positions of each vehicle (loc(t)), the set of vehicle speeds (λ(t)), the amount of data collected by the vehicle (D(t)), the vehicle coverage area (Area(t)), the vehicle's computing power (u(t)), and the perceived utility (PU) are represented as the state. Action: Action a(t) includes fusion method selection F(t), fusion mode selection M(t), and cooperative vehicle selection A(t), which can be represented as follows: Reward: The reward function is associated with the objective function r(t)=ω1G[F(t),M(t),A(t)]-ω2T[F(t),M(t),A(t)].

[0147] Meanwhile, considering the randomness in the training process of DQN model optimization iteration, which may slow down the model training, federated learning technology was added, in which collaborative vehicles train the global DQN model issued by RSU locally.

[0148] In other embodiments, the original optimization problem can be decomposed into: (P1) a perceptual fusion subproblem (joint optimization of cooperative vehicle selection, fusion method selection, and fusion mode selection) with a discrete action subspace; and (P2) a resource allocation subproblem (joint optimization of channel allocation and power control) with a continuous action subspace. The P1 subproblem can be solved using deep reinforcement learning (e.g., D3QN (Dueling Double DQN)) similar to the above. Then, the P2 subproblem can be solved using a joint resource allocation method based on convex optimization theory. For example, the coordinate descent method can be used to divide the variables into transmit power and channel bandwidth. First, the optimal solution is obtained by mathematical transformation for the power control subproblem; second, the channel allocation subproblem is solved using the Gorobi optimization tool; and finally, global optimization is performed based on the coordinate descent method to find the minimum system delay. This disclosure uses D3QN to solve the discrete action space problem, addressing the potential overestimation of Q-values ​​in DQN. Furthermore, this disclosure further uses the block coordinate descent method to decouple the complex resource allocation problem into bandwidth allocation and power allocation subproblems. After decomposition, solving each subproblem becomes a convex optimization problem, which reduces complexity and simplifies the solution.

[0149] As an example, the solution process for this optimization problem is described in detail below.

[0150] C1. As shown in Algorithm 3 below, the optimization problem can be solved by using the block coordinate descent method to decompose the original optimization problem into a perception fusion subproblem with a discrete action subspace and a resource allocation subproblem with a continuous action subspace. First, given the initial values ​​of bandwidth and transmit power, D3QN is used to make decisions on the fusion method selection F, the cooperative vehicle selection matrix A, and the sensor mode selection M.

[0151]

[0152] C2. For the P1 subproblem, perform time-dimensional MDP modeling, defining the state as follows: At time slot t, environmental information including vehicle positions loc(t), vehicle speed set λ(l), vehicle data collection volume D(t), vehicle coverage area Area(t), and vehicle computing power u(t) are represented as the state, expressed as... in This represents a set of multiple state parameters; Action: Action a(t) includes fusion method selection F(t), fusion mode selection M(t), and cooperative vehicle selection A(t), which can be represented as... in Represents a set of multiple state parameters; Reward: The reward function is associated with the objective function r(t)=ω1G[F(t),M(t),A(t)]-ω2T[F(t),M(t),A(t)].

[0153] Considering the overestimation problem of DQN, D3QN can be introduced to solve the problem. D3QN is based on DQN and adds two improvements, double and dueling, to generate discrete action a(t).

[0154] C3. In solving the resource allocation subproblem P2, the coordinate descent method is used to divide the variables into transmit power P and channel bandwidth B. First, the optimal solution for the power control subproblem can be obtained by mathematical transformation. Second, since the bandwidth allocation subproblem is a convex optimization problem, the Gurobi optimization tool (which supports various optimization problems such as linear programming (LP), mixed-integer linear programming (MILP), quadratic programming (QP), mixed-integer quadratic programming (MIQP), quadratic constrained programming (QCP), and mixed-integer quadratic constrained programming (MIQCP)) is used to solve the subproblem. Finally, global optimization is performed based on the coordinate descent method to find the minimum system delay.

[0155] C4. The optimal solutions for bandwidth and transmit power obtained in step C3 are then substituted into C2 for iterative optimization until convergence.

[0156] The following will introduce application examples based on this disclosure.

[0157] The technology disclosed herein can be applied to a variety of products.

[0158] The vehicle-side electronic devices disclosed herein can be implemented, for example, using terminal devices, and the RSU disclosed herein can be implemented, for example, using roadside units (RSUs) in a vehicle network.

[0159] For example, the terminal device can be implemented as a mobile terminal (such as a smartphone, tablet PC, laptop PC, portable gaming terminal, portable / dongle-type mobile router, and digital camera device) or an in-vehicle terminal (such as a car navigation device). The terminal device can also be implemented as a terminal performing machine-to-machine (M2M) communication (also known as a machine-type communication (MTC) terminal). Furthermore, the terminal device can be a wireless communication module (such as an integrated circuit module comprising a single chip) installed on each of the aforementioned terminals.

[0160] [Application examples of RSU]

[0161] (First application example)

[0162] Figure 7 This is a block diagram illustrating a first example of a schematic configuration of an RSU to which the techniques of this disclosure can be applied. The RSU 800 includes one or more antennas 810 and a roadside device 820. The roadside device 820 and each antenna 810 can be connected to each other via RF cables.

[0163] Each of the antennas 810 includes one or more antenna elements (such as multiple antenna elements included in a multiple-input multiple-output (MIMO) antenna) and is used by the roadside equipment 820 to transmit and receive wireless signals. Figure 7 As shown, the RSU 800 may include multiple antennas 810. For example, the multiple antennas 810 may be compatible with multiple frequency bands used by the RSU 800. The roadside device 820 includes a controller 821, a memory 822, a network interface 823, and a wireless communication interface 825.

[0164] The controller 821 can be, for example, a CPU or a DSP, and operates various higher-level functions of the roadside equipment 820. For example, the controller 821 generates data packets based on data in signals processed by the wireless communication interface 825, and transmits the generated packets via the network interface 823. The controller 821 can bundle data from multiple baseband processors to generate bundled packets and transmit the generated bundled packets. The controller 821 may have logical functions that perform controls such as radio resource control, radio bearer control, mobility management, admission control, and scheduling. This control can be performed in conjunction with nearby RSUs or core network nodes (e.g., Access and Mobility Management Functions (AMF)). The memory 822 includes RAM and ROM, and stores programs executed by the controller 821 and various types of control data (such as terminal lists, transmission power data, and scheduling data).

[0165] Network interface 823 is a communication interface for connecting roadside equipment 820 to the core network 824. Controller 821 can communicate with core network nodes or other RSUs via network interface 823. In this case, RSU 800 can be connected to core network nodes or other RSUs via logical interfaces (such as C-V2X uu interface, C-V2X PC5). Network interface 823 can also be a wired communication interface or a wireless communication interface for wireless backhaul. If network interface 823 is a wireless communication interface, it can use a higher frequency band for wireless communication compared to the frequency band used by wireless communication interface 825.

[0166] The wireless communication interface 825 supports any cellular communication scheme and provides wireless connectivity to terminals located in the cell of RSU 800 via antenna 810. The wireless communication interface 825 typically includes, for example, a baseband (BB) processor 826 and RF circuitry 827. The BB processor 826 can perform, for example, encoding / decoding, modulation / demodulation, and multiplexing / demultiplexing, and performs various types of signal processing at layers such as L1, Media Access Control (MAC), Radio Link Control (RLC), and Packet Data Convergence Protocol (PDCP). Instead of controller 821, the BB processor 826 may have some or all of the above-described logical functions. The BB processor 826 may be a memory storing communication control programs, or a module including a processor and associated circuitry configured to execute programs. Update programs can change the functionality of the BB processor 826. The module may be a card or blade inserted into a slot in the roadside equipment 820. Alternatively, the module may be a chip mounted on a card or blade. Meanwhile, the RF circuit 827 may include, for example, a mixer, a filter, and an amplifier, and transmits and receives wireless signals via the antenna 810.

[0167] like Figure 7 As shown, the wireless communication interface 825 may include multiple BB processors 826. For example, the multiple BB processors 826 may be compatible with multiple frequency bands used by the RSU 800. Figure 7 As shown, the wireless communication interface 825 may include multiple RF circuits 827. For example, the multiple RF circuits 827 may be compatible with multiple antenna elements. Although Figure 7 An example is shown in which the wireless communication interface 825 includes multiple BB processors 826 and multiple RF circuits 827, but the wireless communication interface 825 may also include a single BB processor 826 or a single RF circuit 827.

[0168] (Second application example)

[0169] Figure 8 This is a block diagram illustrating a second example of a schematic configuration of an RSU to which the techniques of this disclosure can be applied. The RSU includes one or more antennas 840, roadside equipment 850, and RRH 860. The RRH 860 and each antenna 840 can be connected to each other via RF cables. The roadside equipment 850 and RRH 860 can be connected to each other via high-speed lines such as fiber optic cables.

[0170] Each of the antennas 840 includes one or more antenna elements (such as multiple antenna elements included in a MIMO antenna) and is used by the RRH 860 to transmit and receive wireless signals. Figure 8As shown, the RSU 830 may include multiple antennas 840. For example, the multiple antennas 840 may be compatible with multiple frequency bands used by the RSU 830. The roadside equipment 850 includes a controller 851, a memory 852, a network interface 853, a wireless communication interface 855, and a connection interface 857. The controller 851, memory 852, and network interface 853 are consistent with the reference... Figure 7 The controller 821, memory 822, and network interface 823 described are the same.

[0171] The wireless communication interface 855 supports any cellular communication scheme and provides wireless communication to terminals located in the sector corresponding to the RRH 860 via the RRH 860 and antenna 840. The wireless communication interface 855 may typically include, for example, a BB processor 856. In addition to the BB processor 856 being connected to the RF circuitry 864 of the RRH 860 via a connection interface 857, the BB processor 856 is connected to the reference... Figure 7 The described BB processor 826 is the same. Figure 8 As shown, the wireless communication interface 855 may include multiple BB processors 856. For example, the multiple BB processors 856 may be compatible with multiple frequency bands used by the RSU 830. Although Figure 8 An example is shown in which the wireless communication interface 855 includes multiple BB processors 856, but the wireless communication interface 855 may also include a single BB processor 856.

[0172] Connection interface 857 is an interface for connecting roadside equipment 850 (wireless communication interface 855) to RRH 860. Connection interface 857 can also be a communication module for connecting roadside equipment 850 (wireless communication interface 855) to the aforementioned high-speed line RRH 860.

[0173] The RRH 860 includes a connectivity interface 861 and a wireless communication interface 863.

[0174] Connection interface 861 is an interface for connecting RRH 860 (wireless communication interface 863) to roadside equipment 850. Connection interface 861 can also be a communication module for communication in the aforementioned high-speed line.

[0175] The wireless communication interface 863 transmits and receives wireless signals via antenna 840. The wireless communication interface 863 typically includes, for example, RF circuitry 864. RF circuitry 864 may include, for example, a mixer, a filter, and an amplifier, and transmits and receives wireless signals via antenna 840. Figure 8 As shown, the wireless communication interface 863 may include multiple RF circuits 864. For example, the multiple RF circuits 864 may support multiple antenna elements. Although Figure 8An example is shown in which the wireless communication interface 863 includes multiple RF circuits 864, but the wireless communication interface 863 may also include a single RF circuit 864.

[0176] exist Figure 7 and Figure 8 In the RSU 800 and RSU 830 shown, reference Figure 2 The described processing circuit 210 and reference Figure 4 One or more components included in the described processing circuitry 410 may be implemented in the wireless communication interface 912. Alternatively, at least some of these components may also be implemented by controllers 821 and 851.

[0177] [Application examples of terminal devices]

[0178] (First application example)

[0179] Figure 9 This is a block diagram illustrating an example of a schematic configuration of a smartphone 900 to which the technologies of this disclosure can be applied. The smartphone 900 includes a processor 901, a memory 902, a storage device 903, an external connection interface 904, a camera device 906, a sensor 907, a microphone 908, an input device 909, a display device 910, a speaker 911, a wireless communication interface 912, one or more antenna switches 915, one or more antennas 916, a bus 917, a battery 918, and an auxiliary controller 919.

[0180] The processor 901 can be, for example, a CPU or a system-on-a-chip (SoC), and controls the application layer and other functions of the smartphone 900. The memory 902 includes RAM and ROM, and stores data and programs executed by the processor 901. The storage device 903 can include storage media such as semiconductor memory and hard disks. The external connectivity interface 904 is an interface for connecting external devices, such as memory cards and Universal Serial Bus (USB) devices, to the smartphone 900.

[0181] The camera device 906 includes an image sensor (such as a charge-coupled device (CCD) and complementary metal-oxide-semiconductor (CMOS)) and generates captured images. The sensor 907 may include a set of sensors, such as a measurement sensor, a gyroscope sensor, a magnetometer sensor, and an accelerometer sensor. The microphone 908 converts sound input to the smartphone 900 into an audio signal. The input device 909 includes, for example, a touch sensor, keypad, keyboard, buttons, or switches configured to detect touches on the screen of the display device 910 and receives operations or information input from the user. The display device 910 includes a screen (such as a liquid crystal display (LCD) and an organic light-emitting diode (OLED) display) and displays the output image of the smartphone 900. The speaker 911 converts the audio signal output from the smartphone 900 into sound.

[0182] The wireless communication interface 912 supports any cellular communication scheme (such as LTE and LTE-Advanced) and performs wireless communication. The wireless communication interface 912 typically includes, for example, a BB processor 913 and RF circuitry 914. The BB processor 913 can perform, for example, encoding / decoding, modulation / demodulation, and multiplexing / demultiplexing, and performs various types of signal processing for wireless communication. Meanwhile, the RF circuitry 914 can include, for example, a mixer, filters, and amplifiers, and transmits and receives wireless signals via an antenna 916. The wireless communication interface 912 can be a single chip module on which the BB processor 913 and RF circuitry 914 are integrated. Figure 9 As shown, the wireless communication interface 912 may include multiple BB processors 913 and multiple RF circuits 914. Although Figure 9 An example is shown in which the wireless communication interface 912 includes multiple BB processors 913 and multiple RF circuits 914, but the wireless communication interface 912 may also include a single BB processor 913 or a single RF circuit 914.

[0183] In addition to cellular communication schemes, the wireless communication interface 912 can support other types of wireless communication schemes, such as short-range wireless communication schemes, near-field communication schemes, and wireless local area network (LAN) schemes. In this case, the wireless communication interface 912 may include a BB processor 913 and RF circuitry 914 for each wireless communication scheme.

[0184] Each of the antenna switches 915 switches the connection destination of the antenna 916 among multiple circuits (e.g., circuits for different wireless communication schemes) included in the wireless communication interface 912.

[0185] Each of the antennas 916 includes one or more antenna elements (such as multiple antenna elements included in a MIMO antenna) and is used by the wireless communication interface 912 to transmit and receive wireless signals. Figure 9 As shown, the smartphone 900 may include multiple antennas 916. Although Figure 9 An example is shown in which the smartphone 900 includes multiple antennas 916, but the smartphone 900 may also include a single antenna 916.

[0186] Furthermore, the smartphone 900 may include an antenna 916 for each wireless communication scheme. In this case, the antenna switch 915 can be omitted from the configuration of the smartphone 900.

[0187] Bus 917 connects processor 901, memory 902, storage device 903, external connection interface 904, camera device 906, sensor 907, microphone 908, input device 909, display device 910, speaker 911, wireless communication interface 912, and auxiliary controller 919 to each other. Battery 918 supplies power to... Figure 9 The various blocks of the smartphone 900 shown are powered, and the feeders are partially shown as dashed lines in the figure. The auxiliary controller 919 operates the minimum necessary functions of the smartphone 900, for example, in sleep mode.

[0188] exist Figure 9 In the smartphone 900 shown, reference Figure 2 The described processing circuit 210 and reference Figure 4 One or more components included in the described processing circuitry 410 may be implemented in the wireless communication interface 912. Alternatively, at least some of these components may also be implemented by the processor 901 or the auxiliary controller 919.

[0189] (Second application example)

[0190] Figure 10 This is a block diagram illustrating an example of a schematic configuration of a car navigation device 920 to which the technology of this disclosure can be applied. The car navigation device 920 includes a processor 921, a memory 922, a Global Positioning System (GPS) module 924, a sensor 925, a data interface 926, a content player 927, a storage medium interface 928, an input device 929, a display device 930, a speaker 931, a wireless communication interface 933, one or more antenna switches 936, one or more antennas 937, and a battery 938.

[0191] The processor 921 can be, for example, a CPU or a SoC, and controls the navigation functions and other functions of the car navigation device 920. The memory 922 includes RAM and ROM, and stores data and programs executed by the processor 921.

[0192] GPS module 924 uses GPS signals received from GPS satellites to measure the location (such as latitude, longitude, and altitude) of car navigation device 920. Sensor 925 may include a set of sensors, such as a gyroscope sensor, a geomagnetic sensor, and an air pressure sensor. Data interface 926 is connected to, for example, an in-vehicle network 941 via a terminal not shown, and acquires data generated by the vehicle (such as vehicle speed data).

[0193] Content player 927 reproduces content stored on storage media (such as CDs and DVDs), which is inserted into storage media interface 928. Input device 929 includes, for example, a touch sensor, button, or switch configured to detect touch on the screen of display device 930, and receives operations or information input from the user. Display device 930 includes a screen such as an LCD or OLED display and displays images or reproduced content for navigation functions. Speaker 931 outputs sound for navigation functions or reproduced content.

[0194] The wireless communication interface 933 supports any cellular communication scheme (such as LTE and LTE-Advanced) and performs wireless communication. The wireless communication interface 933 typically includes, for example, a BB processor 934 and RF circuitry 935. The BB processor 934 can perform, for example, encoding / decoding, modulation / demodulation, and multiplexing / demultiplexing, and performs various types of signal processing for wireless communication. Meanwhile, the RF circuitry 935 can include, for example, a mixer, filters, and amplifiers, and transmits and receives wireless signals via an antenna 937. The wireless communication interface 933 can also be a chip module on which the BB processor 934 and RF circuitry 935 are integrated. Figure 10 As shown, the wireless communication interface 933 may include multiple BB processors 934 and multiple RF circuits 935. Although Figure 10 An example is shown in which the wireless communication interface 933 includes multiple BB processors 934 and multiple RF circuits 935, but the wireless communication interface 933 may also include a single BB processor 934 or a single RF circuit 935.

[0195] In addition to cellular communication schemes, the wireless communication interface 933 can support other types of wireless communication schemes, such as short-range wireless communication schemes, near-field communication schemes, and wireless LAN schemes. In this case, for each wireless communication scheme, the wireless communication interface 933 may include a BB processor 934 and an RF circuit 935.

[0196] Each of the antenna switches 936 switches the connection destination of the antenna 937 among multiple circuits (such as circuits for different wireless communication schemes) included in the wireless communication interface 933.

[0197] Each of the antennas 937 includes one or more antenna elements (such as multiple antenna elements included in a MIMO antenna) and is used by the wireless communication interface 933 to transmit and receive wireless signals. Figure 10 As shown, the car navigation device 920 may include multiple antennas 937. Although Figure 10 An example is shown in which the car navigation device 920 includes multiple antennas 937, but the car navigation device 920 may also include a single antenna 937.

[0198] Furthermore, the car navigation device 920 may include an antenna 937 for each wireless communication scheme. In this case, the antenna switch 936 can be omitted from the configuration of the car navigation device 920.

[0199] Battery 938 via feeder to Figure 10 The various blocks of the car navigation device 920 shown are powered, and the feeders are partially shown as dashed lines in the figure. Battery 938 accumulates the power supplied from the vehicle.

[0200] exist Figure 10 In the car navigation device 920 shown, reference Figure 2 The described processing circuit 210 and reference Figure 4 One or more components included in the described processing circuitry 410 may be implemented in the wireless communication interface 912. Alternatively, at least some of these components may also be implemented by the processor 921.

[0201] The technology disclosed herein can also be implemented as an in-vehicle system (or vehicle) 940 comprising one or more of the following blocks: a car navigation device 920, an in-vehicle network 941, and a vehicle module 942. The vehicle module 942 generates vehicle data (such as vehicle speed, engine speed, and fault information) and outputs the generated data to the in-vehicle network 941.

[0202] It should be understood that the reference to "embodiment" or similar expressions in this specification means that a specific feature, structure, or characteristic described in connection with that embodiment is included in at least one specific embodiment of this disclosure. Therefore, the appearance of the terms "in embodiments of this disclosure" and similar expressions in this specification does not necessarily refer to the same embodiment.

[0203] Those skilled in the art will understand that this disclosure can be implemented as a system, apparatus, method, or as a computer-readable storage medium (e.g., a non-transient storage medium) as a computer program product. Therefore, this disclosure can be implemented in various forms, such as a completely hardware embodiment, a completely software embodiment (including firmware, resident software, microprogram code, etc.), or a software and hardware embodiment, hereinafter referred to as a "circuit," "module," or "system." Furthermore, this disclosure can also be implemented as a computer program product in any tangible media form, having computer-usable program code stored thereon.

[0204] The description herein is based on flowcharts and / or block diagrams of systems, apparatuses, methods, and computer program products according to specific embodiments of this disclosure. It will be understood that each block in each flowchart and / or block diagram, and any combination of blocks in the flowcharts and / or block diagrams, can be implemented using computer program instructions. These computer program instructions are executable by a machine comprising a processor of a general-purpose computer or a special-purpose computer, or other programmable data processing means, and are processed by the computer or other programmable data processing means to perform the functions or operations described in the flowcharts and / or block diagrams.

[0205] The accompanying drawings illustrate flowcharts and block diagrams showing the architecture, functionality, and operation of systems, apparatuses, methods, and computer program products achievable according to various embodiments of the present disclosure. Thus, each block in a flowchart or block diagram may represent a module, segment, or portion of program code, including one or more executable instructions to implement a specified logical function. It should also be noted that in some other embodiments, the functions described in a block may not be performed in the order shown in the figures. For example, two blocks illustrated as connected may actually be executed simultaneously, or in some cases, depending on the functions involved, they may be executed in the reverse order shown in the figures. Furthermore, it should be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, may be implemented by a system based on dedicated hardware, or by a combination of dedicated hardware and computer instructions, to perform specific functions or operations.

[0206] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to market technology of the embodiments, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0207] Note that the technology disclosed in this specification may have the following configurations.

[0208] (1) An electronic device for use on a first device side, the electronic device comprising:

[0209] Processing circuit, the processing circuit being configured to:

[0210] Receive model information from the collaborative sensing model of the second device;

[0211] Based on the model information of the collaborative sensing model of the second device, determine the differences between the collaborative sensing model of the second device and the collaborative sensing model of the first device; and

[0212] Based on the differences, a collaborative sensing strategy for the first device and the second device is determined.

[0213] (2) The electronic device according to (1), wherein,

[0214] The model information of the collaborative sensing model of the second device includes at least one of the following: the structure and parameter information of the intermediate fusion layer of the collaborative sensing model, the modal information of the source data processed by the base layer of the collaborative sensing model, and the information of the training dataset used to train the collaborative sensing model.

[0215] (3) The electronic device according to (1), wherein,

[0216] The model information of the collaborative sensing model of the second device includes the sensing capability indicators of the collaborative sensing model.

[0217] The perception capability index is used to indicate one or more of the following: depth perception capability, color perception capability, shape perception capability, size perception capability, object category recognition capability, perception range size, and perception accuracy.

[0218] (4) The electronic device according to (1), wherein,

[0219] The model information of the collaborative sensing model of the second device includes intermediate feature values ​​generated by the collaborative sensing model for a specific dataset.

[0220] The specific dataset includes one or more of the following: historical datasets processed by the collaborative sensing model of the second device, and benchmark datasets indicated by the first device to the second device.

[0221] (5) According to the electronic device described in (1), the processing circuit is further configured to:

[0222] A collaborative sensing service request is sent to the second device, so that the second device responds to the collaborative sensing service request and sends the model information to the first device.

[0223] (6) According to the electronic device described in (1), the processing circuit is further configured to:

[0224] Receive a retraining request for retraining the collaborative sensing model of the second device;

[0225] In response to the retraining request, a dataset for retraining is provided to the second device.

[0226] (7) The electronic device according to (1), wherein,

[0227] The second device includes multiple cooperative sensing models, and the cooperative sensing strategy indicates which of the multiple cooperative sensing models is used for cooperative sensing with the first device; and / or

[0228] The second device includes a plurality of second devices, and the cooperative sensing strategy indicates which of the plurality of second devices is used to perform cooperative sensing with the first device.

[0229] (8) According to the electronic device described in (1), the processing circuit is further configured to:

[0230] The system receives collaborative sensing messages from multiple second devices, each indicating one or more of the following: the location, speed, computing power, data transmission volume, communication channel conditions, type of onboard sensors, sensing range, and data transmission volume of the second devices; and

[0231] The collaborative sensing strategy is determined based on the collaborative sensing target of the first device and the collaborative sensing messages received from the plurality of second devices.

[0232] (9) The electronic device according to (8), wherein,

[0233] The collaborative sensing objective of the first device is related to one or more of the following: collaborative sensing task matching degree, communication channel bandwidth constraint, transmit power constraint, and system latency.

[0234] The collaborative sensing strategy indicates one or more of the following: a second device among the plurality of second devices for collaborative sensing with the first device, a fusion method of the collaborative sensing models of the first device and the second device for collaborative sensing, and a fusion mode of the data modes of the first device and the second device for collaborative sensing.

[0235] (10) The electronic device according to (1), wherein,

[0236] The cooperative perception strategy is determined through Deep Q-Network (DQN) and / or federated learning.

[0237] (11) An electronic device for a second device side, the electronic device comprising:

[0238] Processing circuit, the processing circuit being configured to:

[0239] The system provides a first device with model information of the collaborative sensing model of the second device, so that the first device can determine the differences between the collaborative sensing model of the second device and the collaborative sensing model of the first device based on the model information of the collaborative sensing model of the second device, and determine a collaborative sensing strategy for the first device and the second device based on the differences; and

[0240] Receive the collaborative sensing strategy from the first device.

[0241] (12) The electronic device according to (11), wherein,

[0242] The model information of the collaborative sensing model of the second device includes at least one of the following: the structure and parameter information of the intermediate fusion layer of the collaborative sensing model, the modal information of the source data processed by the base layer of the collaborative sensing model, and the information of the training dataset used to train the collaborative sensing model.

[0243] (13) The electronic device according to (11), wherein,

[0244] The model information of the collaborative sensing model of the second device includes the sensing capability indicators of the collaborative sensing model.

[0245] The perception capability index is used to indicate one or more of the following: depth perception capability, color perception capability, shape perception capability, size perception capability, object category recognition capability, perception range size, and perception accuracy.

[0246] (14) The electronic device according to (11), wherein,

[0247] The model information of the collaborative sensing model of the second device includes intermediate feature values ​​generated by the collaborative sensing model for a specific dataset.

[0248] The specific dataset includes one or more of the following: historical datasets processed by the collaborative sensing model of the second device, and benchmark datasets indicated by the first device to the second device.

[0249] (15) According to the electronic device of (11), the processing circuit is further configured to:

[0250] Receive a collaborative sensing service request from the first device; and

[0251] In response to the collaborative sensing service request, the model information is sent to the first device.

[0252] (16) The electronic device according to (11), wherein,

[0253] In response to the fact that the positions of the second device and the first device satisfy a predetermined positional relationship, the first device is provided with model information of the collaborative sensing model of the second device.

[0254] (17) According to the electronic device of (11), the processing circuit is further configured to:

[0255] Send a retraining request to the first electronic device to retrain the collaborative perception model of the second device.

[0256] (18) The electronic device according to (11), wherein,

[0257] The second device includes multiple cooperative sensing models, and the cooperative sensing strategy indicates which of the multiple cooperative sensing models is used to perform cooperative sensing with the first device.

[0258] (19) According to the electronic device of (11), the processing circuit is further configured to:

[0259] The first electronic device sends a collaborative sensing message of the second device to the first device. The collaborative sensing message indicates one or more of the following: the location, speed, computing power, data transmission volume, communication channel conditions, type of sensor, sensing range, and sensing data volume of the second device, so that the first electronic device can determine the collaborative sensing strategy based on the collaborative sensing target of the first device and the collaborative sensing messages of the plurality of second devices.

[0260] (20) A collaborative sensing method for a first device side, comprising:

[0261] Receive model information from the collaborative sensing model of the second device;

[0262] Based on the model information of the collaborative sensing model of the second device, determine the differences between the collaborative sensing model of the second device and the collaborative sensing model of the first device; and

[0263] Based on the differences, a collaborative sensing strategy for the first device and the second device is determined.

[0264] (21) A collaborative sensing method for a second device side, comprising:

[0265] The system provides a first device with model information of the collaborative sensing model of the second device, so that the first device can determine the differences between the collaborative sensing model of the second device and the collaborative sensing model of the first device based on the model information of the collaborative sensing model of the second device, and determine a collaborative sensing strategy for the first device and the second device based on the differences; and

[0266] Receive the collaborative sensing strategy from the first device.

[0267] (22) A computer-readable storage medium including executable instructions that, when executed by an information processing device, cause the information processing device to perform the cooperative sensing method according to (20) or (21).

[0268] (23) A computer program product comprising a computer program that, when executed by a processor, causes the processor to perform the collaborative sensing method according to (20) or (21).

Claims

1. An electronic device for a first device side, the electronic device comprising: Processing circuit, the processing circuit being configured to: Receive model information from the collaborative sensing model of the second device; Based on the model information of the collaborative sensing model of the second device, determine the differences between the collaborative sensing model of the second device and the collaborative sensing model of the first device; and Based on the differences, a collaborative sensing strategy for the first device and the second device is determined.

2. The electronic device according to claim 1, wherein, The model information of the collaborative sensing model of the second device includes at least one of the following: the structure and parameter information of the intermediate fusion layer of the collaborative sensing model, the modal information of the source data processed by the base layer of the collaborative sensing model, and the information of the training dataset used to train the collaborative sensing model.

3. The electronic device according to claim 1, wherein, The model information of the collaborative sensing model of the second device includes the sensing capability indicators of the collaborative sensing model. The perception capability index is used to indicate one or more of the following: depth perception capability, color perception capability, shape perception capability, size perception capability, object category recognition capability, perception range size, and perception accuracy.

4. The electronic device according to claim 1, wherein, The model information of the collaborative sensing model of the second device includes intermediate feature values ​​generated by the collaborative sensing model for a specific dataset. The specific dataset includes one or more of the following: historical datasets processed by the collaborative sensing model of the second device, and benchmark datasets indicated by the first device to the second device.

5. The electronic device according to claim 1, wherein the processing circuit is further configured to: A collaborative sensing service request is sent to the second device, so that the second device responds to the collaborative sensing service request and sends the model information to the first device.

6. The electronic device according to claim 1, wherein the processing circuit is further configured to: Receive a retraining request for retraining the collaborative sensing model of the second device; In response to the retraining request, a dataset for retraining is provided to the second device.

7. The electronic device according to claim 1, wherein, The second device includes multiple cooperative sensing models, and the cooperative sensing strategy indicates which of the multiple cooperative sensing models is used for cooperative sensing with the first device; and / or The second device includes a plurality of second devices, and the cooperative sensing strategy indicates which of the plurality of second devices is used to perform cooperative sensing with the first device.

8. The electronic device according to claim 1, wherein the processing circuit is further configured to: The system receives collaborative sensing messages from multiple second devices, each indicating one or more of the following: the location, speed, computing power, data transmission volume, communication channel conditions, type of onboard sensors, sensing range, and data transmission volume of the second devices; and The collaborative sensing strategy is determined based on the collaborative sensing target of the first device and the collaborative sensing messages received from the plurality of second devices.

9. The electronic device according to claim 8, wherein, The collaborative sensing objective of the first device is related to one or more of the following: collaborative sensing task matching degree, communication channel bandwidth constraint, transmit power constraint, and system latency. The collaborative sensing strategy indicates one or more of the following: a second device among the plurality of second devices for collaborative sensing with the first device, a fusion method of the collaborative sensing models of the first device and the second device for collaborative sensing, and a fusion mode of the data modes of the first device and the second device for collaborative sensing.

10. The electronic device according to claim 1, wherein, The cooperative perception strategy is determined through Deep Q-Network (DQN) and / or federated learning.

Images