Communication equipment deployment method and device, computer equipment and readable storage medium

Abstract

The embodiment of the invention discloses a communication equipment deployment method and device, computer equipment and a readable storage medium. The method comprises the following steps: determining a target deployment performance index meeting a target deployment requirement; processing an input data source based on the target deployment performance index by analyzing a neural network, so as to extract fusion features meeting the target deployment performance index from the input data source; and processing the fusion feature through a reinforcement learning network, and determining a target deployment scheme of the communication device in the target area, so that the accumulated reward of the reinforcement learning network determined based on the target deployment performance index is maximized.

Description

Technical Field

[0001] The present disclosure relates to the field of communication and computer technology, and in particular to a communication equipment deployment method, a communication equipment deployment apparatus, a computer device, and a computer-readable storage medium. Background Art

[0002] The communication equipment deployment solutions provided in the related art are often targeted at a certain deployment requirement. When the deployment requirement changes, a new deployment solution needs to be redesigned, which reduces the versatility and flexibility of the deployment solution and increases the design cost and time of the deployment solution. Summary of the invention

[0003] The embodiments of the present disclosure provide a communication equipment deployment method, a communication equipment deployment apparatus, a computer device, and a computer-readable storage medium, which can improve the versatility and flexibility of communication equipment deployment and reduce the design cost and time of deployment solutions.

[0004] An embodiment of the present disclosure provides a communication equipment deployment method, the method comprising: determining a target deployment performance indicator that meets the target deployment requirements; processing an input data source based on the target deployment performance indicator by analyzing a neural network to extract fusion features that meet the target deployment performance indicator from the input data source; processing the fusion features by a reinforcement learning network to determine a target deployment scheme for the communication equipment in a target area, so as to maximize the cumulative reward of the reinforcement learning network determined based on the target deployment performance indicator.

[0005] An embodiment of the present disclosure provides a communication equipment deployment device, which includes: a determination unit, used to determine a target deployment performance indicator that meets the target deployment requirements; a processing unit, used to process an input data source based on the target deployment performance indicator by analyzing a neural network, so as to extract a fusion feature that meets the target deployment performance indicator from the input data source; the processing unit is also used to process the fusion feature through a reinforcement learning network, and determine a target deployment scheme of the communication equipment in a target area, so as to maximize the cumulative reward of the reinforcement learning network determined based on the target deployment performance indicator.

[0006] An embodiment of the present disclosure provides a computer device, including a processor, a memory, and an input / output interface; the processor is connected to the memory and the input / output interface, respectively, wherein the input / output interface is used to receive data and output data, the memory is used to store a computer program, and the processor is used to call the computer program so that the computer device including the processor executes the communication device deployment method in any embodiment of the present disclosure.

[0007] An embodiment of the present disclosure provides a computer-readable storage medium storing a computer program, wherein the computer program is suitable for being loaded and executed by a processor so that a computer device having the processor executes the communication device deployment method in any embodiment of the present disclosure.

[0008] The embodiments of the present disclosure provide a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional modes in any embodiment of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0010] Figure 1 It is a flowchart of a communication equipment deployment method provided by an embodiment of the present disclosure.

[0011] Figure 2 It is a schematic diagram of a universal RSU (Roadside Unit) deployment framework provided by an embodiment of the present disclosure.

[0012] Figure 3 It is a schematic diagram of intelligent modeling of RSU deployment provided by an embodiment of the present disclosure.

[0013] Figure 4 It is a functional schematic diagram of a fusion network provided by an embodiment of the present disclosure.

[0014] Figure 5 It is a structural diagram of a fusion network provided by an embodiment of the present disclosure.

[0015] Figure 6 It is a schematic diagram of an RSU deployment process based on MDP (Markov Decision Process) provided in an embodiment of the present disclosure.

[0016] Figure 7 It is a network architecture diagram of an RSU deployment framework based on DRL (Deep Reinforcement Learning) provided in an embodiment of the present disclosure.

[0017] Figure 8 It is a schematic diagram of an action space initialization filtering operation provided by an embodiment of the present disclosure.

[0018] Fig. 9 It is a structural diagram of a communication equipment deployment device provided in an embodiment of the present disclosure.

[0019] Fig.10 It is a network interaction architecture diagram of a communication device deployment method provided by an embodiment of the present disclosure.

[0020] Fig.11 It is a structural diagram of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0022] It should be understood that the "several" mentioned in this article refers to one or more, and "multiple" refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0023] Figure 1 FIG. 1 is a flow chart of a communication device deployment method provided by an embodiment of the present disclosure. Figure 1 As shown, the method provided by the embodiment of the present disclosure may include the following steps. The communication device in the embodiment of the present disclosure may be any electronic device that has deployment requirements and has the ability to send and / or receive signals.

[0024] In S110 , a target deployment performance indicator that meets the target deployment requirement is determined.

[0025] In the disclosed embodiment, the target deployment requirement can be determined according to the target area (region or city) where the communication device is currently to be deployed and the determined deployment task. When the target area and / or deployment task changes, the target deployment requirement changes accordingly. The target deployment performance index is a performance index used to measure whether the determined target deployment requirement is met. The target deployment performance index can change with the change of the target deployment requirement.

[0026] In an exemplary embodiment, determining a target deployment performance indicator that meets a target deployment requirement includes: providing an expandable output indicator library, wherein the output indicator library includes various types of deployment performance indicators; and selecting a target deployment performance indicator that can meet the target deployment requirement from the deployment performance indicators in the output indicator library.

[0027] In the embodiments of the present disclosure, an expandable output indicator library is provided, that is, the output indicator library may include various types of deployment performance indicators. The types, number of types, indicator weights, etc. of the deployment performance indicators included in the output indicator library can be increased, reduced, and updated according to actual needs. Therefore, no matter how the proposed target deployment requirements change, the target deployment performance indicators that meet the target deployment requirements can be obtained from the output indicator library.

[0028] In S120, the input data source is processed based on the target deployment performance indicator by analyzing the neural network to extract fusion features that meet the target deployment performance indicator from the input data source.

[0029] The input data source in the embodiment of the present disclosure refers to any data source related to the deployment of communication equipment, such as any one or more of the map of the target area, the road distribution on the map, the traffic conditions, the deployment location of the communication equipment already deployed in the target area, etc. The analytical neural network in the embodiment of the present disclosure can be used to automatically screen each input data source according to the selected target deployment performance index, automatically select the basic features of the input data source that can be used to meet the target deployment performance index, and fuse the basic features of the selected input data source that can be used to meet the deployment performance index to obtain the fusion features that are effective for meeting the target deployment requirements, and eliminate or weaken the basic features of the input data source that are invalid or inefficient for meeting the target deployment requirements. There is no need to manually select the input data source, and there is no need to manually design the features extracted from the input data source. At the same time, when the target deployment requirements change, there is no need to redesign the model for deploying communication equipment and retrain the parameters of the model.

[0030] In an exemplary embodiment, the method provided by the embodiment of the present disclosure further includes: setting the indicator weight of each target deployment performance indicator according to the target deployment requirement. Wherein, the input data source is processed based on the target deployment performance indicator by analyzing the neural network to extract the fusion feature that meets the target deployment performance indicator from the input data source, including: the input data source is processed based on the target deployment performance indicator and its indicator weight by analyzing the neural network to extract the fusion feature that meets the target deployment performance indicator and its indicator weight from the input data source.

[0031] Optionally, the indicator weight of each target deployment performance indicator can also be set according to the target deployment requirements to reflect the emphasis on different target deployment performance indicators in the target deployment requirements. When the target deployment requirements change, the type, quantity and / or performance indicator values ​​of the selected target deployment performance indicators can be adaptively changed. When the target deployment performance indicator and / or its performance indicators change, the analysis neural network can automatically adjust the fusion features obtained from the input data source to meet the changing target deployment requirements.

[0032] In an exemplary embodiment, the method provided by the embodiment of the present disclosure further includes: providing an expandable input information library, wherein the input information library includes various types of input data sources.

[0033] The embodiments of the present disclosure provide an expandable input information library, that is, the type and / or quantity of input data sources contained in the input information library can be changed, so that the input information library can contain input data sources that can reflect various deployment requirements, so that when the target deployment requirements change, effective fusion features can still be obtained from the input information library.

[0034] In S130, the fusion feature is processed by a reinforcement learning network to determine a target deployment scheme of the communication device in a target area, so as to maximize a cumulative reward of the reinforcement learning network determined based on the target deployment performance indicator.

[0035] In the disclosed embodiment, by jointly analyzing the neural network and the reinforcement learning network, the target deployment plan of the communication device in the target area that is finally outputted can have the maximum cumulative reward, that is, the target deployment plan is globally optimal.

[0036] In an exemplary embodiment, the analysis neural network includes a feature extraction network and a fusion network. The analysis neural network processes the input data source based on the target deployment performance index to extract fusion features that meet the target deployment performance index from the input data source, including: inputting Z input data sources into the feature extraction network to obtain N basic features of the Z input data sources, where Z is a positive integer greater than or equal to 1, and N is a positive integer greater than or equal to Z; splicing the N basic features of the Z input data sources to generate a basic feature set; processing the basic feature set through the fusion network to generate a compressed feature set, and learning a feature weight frame based on the compressed feature set, wherein the feature weight frame includes N feature weights, and each feature weight indicates the importance of the corresponding basic feature; generating the fusion feature based on the feature weight frame and the basic feature set.

[0037] In the disclosed embodiment, N basic features extracted from Z input data sources are processed through a fusion network to learn N feature weights, and the N feature weights may respectively refer to the importance of each basic feature of the N basic features for meeting the target deployment requirements, that is, the basic feature with a larger feature weight is more important for meeting the target deployment requirements, and the corresponding basic feature accounts for a larger proportion in the fused feature; the basic feature with a smaller feature weight accounts for a smaller proportion in the fused feature; the basic feature with a feature weight of 0 will be automatically removed from the fused feature, thereby realizing automatic screening of basic features.

[0038] In an exemplary embodiment, the method provided by the embodiment of the present disclosure also includes: determining the rewards collected by the reinforcement learning network according to the target deployment performance indicator; and jointly training the network parameters of the fusion network and the parameters of the reinforcement learning network based on the collected rewards so that the feature weight frame determined based on the fusion network is associated with the target deployment performance indicator.

[0039] In the disclosed embodiment, the reward collected by the reinforcement learning network is determined based on the target deployment performance indicator (optionally, it may also include its indicator weight), so that when the network parameters of the fusion network and the parameters of the reinforcement learning network are jointly trained based on the reward, the network parameters of the fusion network can reflect the target deployment performance indicator (optionally, it may also include its indicator weight). Therefore, when the feature weights of the basic features of the input data source are processed based on the fusion network with corresponding network parameters, the basic features that are effective in satisfying the target deployment performance indicator (optionally, it may also include its indicator weight) can be automatically selected to generate fusion features.

[0040] In an exemplary embodiment, the reinforcement learning network includes an environment and an agent. The fusion feature is processed by the reinforcement learning network to determine the target deployment scheme of the communication device, including: changing the environment based on the action output by the agent, the action is used to deploy the communication device in the target area; determining the state and reward observed from the environment based on the changed environment, the state observed from the environment includes the fusion feature and the deployed action information of the target area, and the reward is determined according to the target deployment performance indicator; the state and the reward are processed by the agent to output the next action, and the cumulative reward is the accumulation of the rewards generated by each action.

[0041] In the embodiment of the present disclosure, the deployed action information refers to the relevant information of the communication equipment deployed in the target area, such as the deployment location, signal coverage, etc. of the communication equipment deployed in the target area. The method provided in the embodiment of the present disclosure is to incrementally deploy the communication equipment based on the communication equipment deployed in the target area.

[0042] In an exemplary embodiment, the state observed from the environment further includes deployed action coverage information and / or deployed action mask information.

[0043] In the disclosed embodiment, the deployed action coverage information refers to the signal coverage range of the communication equipment that has been deployed in the target area, which can be extracted from the deployed action information. The deployed action mask information refers to the indication of the signal coverage range of the communication equipment that has been deployed in the target area and the range that has not been covered by the signal of the communication equipment on the map of the target area. The more information observed from the environment, the more accurate the action predicted by the intelligent agent, that is, the more the deployment of the communication equipment can reach the global optimum.

[0044] In an exemplary embodiment, the state observed from the environment also includes deployed action mask information. Wherein, the state and the reward are processed by the agent to output the next action, including: generating a mask according to the deployed action mask information, wherein the mask indicates the signal coverage of the communication equipment deployed in the target area; before the agent outputs the next action based on the state and the reward, the mask is used to remove the signal coverage of the communication equipment deployed in the target area. In the method provided by the embodiment of the present disclosure, the agent uses a mask to remove inefficient actions before outputting an action, which can reduce the amount of data processing and improve data processing efficiency, because for the signal coverage range of the communication equipment that has been deployed in the target area, if communication equipment is deployed in these areas, it is inefficient to meet the target deployment requirements.

[0045] In an exemplary embodiment, the method provided by the embodiment of the present disclosure also includes: obtaining a map of the target area; processing the map to determine candidate deployment areas of communication equipment in the map; and filtering actions that the agent can output based on the candidate deployment areas of communication equipment.

[0046] In the embodiments of the present disclosure, the candidate deployment area of ​​communication equipment refers to feasible deployment locations in the target area based on the target deployment requirements. For example, if the target deployment requirement is to deploy RSU on the road to improve the communication quality of connected vehicles, since RSU is usually deployed on the side of the road, then the candidate deployment area of ​​communication equipment is the road in the target area. When the target deployment requirement changes, the candidate deployment area of ​​communication equipment may change accordingly, and the present disclosure does not limit this. The embodiments of the present disclosure pre-filter the deployment actions of communication equipment through the candidate deployment area of ​​communication equipment, and exclude locations where it is impossible to deploy communication equipment, thereby reducing the amount of data processing and improving deployment efficiency and accuracy.

[0047] The communication equipment deployment method provided by the embodiment of the present disclosure, on the one hand, automatically extracts and learns the fusion features that meet the target deployment performance indicators from the input data source by analyzing the neural network, thereby automatically extracting the fusion features that are effective for meeting the target deployment requirements, thereby improving the accuracy and efficiency of the target deployment plan of the communication equipment; on the other hand, the effective fusion features are processed in combination with the reinforcement learning network, which can form positive guidance for the deployment process to achieve maximum satisfaction of the target deployment performance indicators. At the same time, when the target deployment requirements change, the network structure combining the analysis neural network and the reinforcement learning network does not need to be redesigned, thereby improving the versatility and flexibility and reducing the design cost and time of the deployment plan.

[0048] As a communication gateway, RSU plays an indispensable role in realizing the concept of V2X (Vehicle to Everything) by seamlessly connecting various devices and vehicles. RSU can collect information from sensor equipment, traffic infrastructure and surrounding intelligent connected vehicles, upload this information to the V2X platform through wired or wireless channels, and distribute traffic information to relevant vehicles. By widely and effectively deploying RSU on the road network, the efficiency and coverage of information exchange in V2X can be greatly improved, thereby achieving better road services such as traffic control, road safety and informatization. The integration of various sensors gives RSU great potential for application expansion. Therefore, in the following embodiments, the communication device is used as RSU to illustrate the communication device deployment method provided in the embodiments of the present disclosure, but the present disclosure is not limited to this.

[0049] Currently, there is a lack of an RSU deployment architecture that can describe various deployment requirements and corresponding deployment performance indicators in the same framework. The deployment requirements here refer to the specific purpose of deploying RSU. After determining each deployment requirement, corresponding deployment performance indicators are required to measure whether the deployment requirement is met.

[0050] In order to achieve this goal, researchers at home and abroad have actively explored and studied the RSU deployment problem from different angles. First of all, since the effectiveness of RSU deployment usually needs to be measured by one or more deployment performance indicators, such as spatial coverage, traffic coverage, communication quality, etc., the work can be classified based on this. Among them, some work focuses on improving RSU coverage, such as road coverage, service coverage, traffic coverage, etc. Another part of the work focuses on optimizing communication performance, such as maximizing communication performance through the deployment of RSU. In addition, there are some works with different focuses. For example, deploying RSU for traffic prediction. Deploying RSU for traffic prediction means using deployed RSUs to predict the traffic flow on the road. Specific applications include predicting traffic flow within the coverage area based on historical information statistically collected by RSU, and integrating data collected by other RSUs to infer traffic flow on related roads outside the RSU coverage area through training. It can be seen that even in the research stage, the modeling of RSU deployment problems has its own focus. In actual deployment, it is more likely to encounter more complex deployment goals / deployment requirements, and the deployment goals / deployment requirements will change. The above research method cannot be easily coped with, and it will take more manpower and material resources to modify the original RSU deployment model.

[0051] Secondly, since the research in related technologies varies in problem modeling, and accordingly uses different optimization techniques to solve the RSU deployment problem. For example, using 0-1 integer programming to solve RSU deployment, or constructing the RSU deployment problem as a knapsack problem and solving it using a knapsack algorithm. However, actual RSU deployment is a long process, which usually requires the gradual addition of RSUs on the road network. The research in related technologies only provides a complete RSU deployment solution under the current cost constraints, but does not have the ability to expand.

[0052] Therefore, in order to effectively solve the pain points in the above two RSU deployment problems, the embodiments of the present disclosure propose a universal RSU deployment framework. Figure 2 As shown, the architecture of the RSU deployment platform provided by the embodiment of the present disclosure includes a feature analyzer 100 and a deployment scheduler 200 .

[0053] The feature analyzer 100 includes an input information library 110, an analysis neural network 120, and an output indicator library 130. The input information library 110 is used to provide various types of input data sources related to RSU deployment. The output indicator library 130 is used to provide deployment performance indicators for measuring various deployment requirements. The feature analyzer 100 can obtain a target deployment performance indicator that can be used to measure whether the target deployment requirement is met from the deployment performance indicators of the output indicator library 130 according to the currently determined target deployment requirement. Optionally, the indicator weight of each target deployment performance indicator can also be set according to the target deployment requirement. After the feature analyzer 100 obtains each target deployment performance indicator from the output indicator library 130 (optionally, the indicator weight of each target deployment performance indicator can also be obtained), the feature analyzer 100 can use the analysis neural network 120 to extract various basic features from various types of input data sources provided by the input information library 110, and fuse the various extracted basic features according to the obtained target deployment performance indicators (optionally, also according to the corresponding indicator weights), automatically screen out from the various basic features the target basic features that are effective for meeting the target deployment performance indicators (optionally, also including the corresponding indicator weights), that is, meeting the target deployment requirements, and fuse the effective target basic features to obtain the fusion feature 140, so as to implicitly obtain the relationship between various types of input data sources (modeling features) and the target deployment performance indicators corresponding to the target deployment requirements (optionally, also including the corresponding indicator weights). The key issue in the design of this component of the feature analyzer 100 is how to effectively approximate this complex relationship instead of using a series of pre-defined manually defined features. To this end, the embodiment of the present disclosure designs a new neural network structure (i.e., feature analyzer 100), organizes the input data source / deployment performance indicators (also referred to as metric information) into an extensible library (i.e., input information library 110 and output indicator library 130), and then proposes an efficient analysis neural network 120 to automatically learn the feature weights required to meet the target deployment requirements and perform feature fusion based on this to obtain the fused feature 140.

[0054] In the disclosed embodiment, the input of the feature analyzer 100 component includes a set of general data of the target area / target region where RSU needs to be deployed obtained from the input information library 110, such as a map of the target area / target region, collected or simulated GPS (Global Positioning System) data of transportation tools (such as car lights), etc., and also includes a set of target deployment performance indicators obtained from the output indicator library 130, such as road coverage, communication quality, mobility prediction and other requirements. The output of the feature analyzer 100 component is the fused feature (i.e., fused feature 140), which will be used by the next component, i.e., Figure 2 The deployment scheduler 200 in is used to plan the deployment of RSU.

[0055] The GPS data in the embodiments of the present disclosure refers to the GPS positioning data set of the transportation tools (such as vehicles) collected in the geometric map data of the target area / target region, wherein each vehicle trajectory is represented by continuous GPS coordinates and corresponding time nodes. GPS data of a certain date interval for the entire period can be used, for example, the trajectory data of a large number of vehicles in the target area / target region in 2016 can be used during testing, but the present disclosure is not limited thereto.

[0056] The deployment scheduler 200 component can be used to generate different feasible deployment strategies (the reinforcement learning network 210 can be combined with the corresponding reinforcement learning algorithm to find one deployment strategy after another in its exploration process. The deployment strategy refers to the deployment positions of all RSUs to be deployed in the target area / target area), and through continuous comparison and learning, find the most effective deployment strategy for the target area / target area. The key problem to be solved in the design of the deployment scheduler 200 component is how to effectively compare different deployment strategies, that is, to compare the numerical performance of the required target deployment performance indicators under different deployment schemes (deployment strategies). Since there are many possible locations for each RSU, when considering the need to deploy a large number of RSUs, the search space (that is, the combination of possible locations of these RSUs) will be very large. To this end, the embodiment of the present disclosure designs a customized framework based on deep reinforcement learning (DRL) to find excellent deployment strategies in an effective way.

[0057] Continue to refer Figure 2 The input of the deployment scheduler 200 component may include the fusion features 140 learned by the feature analyzer 100, the map of the target area ( Figure 2 The total number of RSUs that need to be deployed in the target area ( Figure 2 ), various target deployment performance indicators (optionally, corresponding indicator weights) obtained from the output indicator library 130, and deployed action information 220. The deployed action information 220 refers to the location of the existing / deployed RSU in the target area before the current RSU is deployed, such as Figure 2 The deployed action information 220 in the is a display of the currently deployed action (one action corresponds to the deployment of one RSU) on the map of the target area. Optionally, the input of the deployment scheduler 200 may also include other domain knowledge such as deployed action coverage information 230 and / or deployed action mask information 240, so as to enhance the environmental information obtained by the reinforcement learning network 210 and improve the accuracy of the deployment plan 300 generated by the deployment scheduler 200.

[0058] In the embodiment of the present disclosure, the deployed action coverage information 230 refers to the coverage information of the currently deployed action, such as the coverage area of ​​the currently deployed RSU (here, the theoretical coverage area of ​​each RSU is taken as an example to simplify the processing process and improve the processing efficiency). The coverage area here refers to the signal coverage area of ​​the RSU. The deployed action mask information 240 refers to the mask (mask) that excludes the relevant action from the map of the target area according to the currently deployed action, such as Figure 2 The white area in the deployed action mask information 240 indicates the area where the deployment of RSU has been excluded at present, because the white area has been covered by the signal of the existing RSU. Therefore, if it is an inefficient action to continue to deploy RSU in the white area in the future, the RSU deployment action in the white area can be removed, and the black area indicates the area where RSU can be deployed at present. These areas have not been covered by the signal of the existing RSU, and the deployment location of RSU can be selected in the black area in the future.

[0059] Figure 2 In an embodiment, the deployment scheduler 200 may include a corresponding neural network ( Figure 2 ), to respectively encode the data after embedding the fused feature 140, the deployed action information 220, the deployed action coverage information 230, and the deployed action mask information 240, to obtain the fused feature encoding information 141, the deployed action encoding information 221, the deployed action coverage encoding information 231, and the deployed action mask encoding information 241. The deployment scheduler 200 also includes a reinforcement learning network 210, and the fused feature encoding information 141 and the deployed action encoding information 221 can be input into the reinforcement learning network 210. Optionally, the deployed action coverage encoding information 231 and / or the deployed action mask encoding information 241 can also be input into the reinforcement learning network 210. The reinforcement learning network 210 processes the fused feature encoding information 141 and the deployed action encoding information 221 (optionally, also including the deployed action coverage encoding information 231 and / or the deployed action mask encoding information 241), and outputs the final deployment strategy, for example Figure 2 The deployment scheme 300 shown in FIG. 3 gives the deployment position of each RSU 320 in the target area 310 according to the overall optimum (ie, the value of the target deployment performance indicator is maximized, or the weighted sum of the target deployment performance indicator and its indicator weight is maximized). Figure 2 It is assumed that the theoretical coverage area of ​​each RSU 320 is a circle 321 with a radius r and a deployment location of the RSU 320 as the center. The value of r can be determined according to the actual situation of the RSU 320, and the present disclosure does not limit this.

[0060] In the actual deployment solution optimization process, such as Figure 2 The framework shown can automatically learn to decide which group or groups of input data sources to use based on the required deployment goals (i.e., the numerical requirements of the target deployment performance indicators selected from the output indicator library 130 (which may involve multiple target deployment performance indicators) and indicator weights) through the designed analysis neural network 120, and automatically explore how to apply these data (including the fusion features 140 automatically learned by the analysis neural network 120) based on the reinforcement learning network 210 to meet the maximization target deployment requirements, that is, to maximize the value of the weighted sum of the target deployment performance indicators and their indicator weights through training, thereby guiding RSU deployment.

[0061] The disclosed embodiment proposes a general RSU intelligent modeling and planning method. The method uses the designed analytical neural network to automatically learn and extract features useful for RSU deployment to generate fusion features, and combines reinforcement learning to form positive guidance for the RSU deployment process to achieve the maximum satisfaction of the deployment goal. Since the disclosed embodiment provides a data input and demand output interface (i.e., a data input interface) that can be expanded according to deployment requirements for RSU deployment for potential networked autonomous driving applications, the RSU deployment of the networked autonomous driving application can be greatly improved. Figure 2 When the deployment requirements change, there is no need to change the input information library 110 and the output indicator library 120. Figure 2 The RSU deployment framework shown only needs to extract the corresponding target deployment performance indicators and their indicator weights from the output indicator library 130, thus providing a general RSU deployment modeling and planning method, thereby improving the efficiency of RSU deployment and reducing the time and cost required for RSU deployment. At the same time, the embodiment of the present disclosure incorporates an input information library composed of various input data sources, such as geometric map data and mobility data (such as GPS data), and an output indicator library including various widely used deployment performance indicators, such as road coverage, traffic coverage, and communication quality indicators, and automatically extracts effective fusion features from various input data sources through the obtained target deployment performance indicators, so that the RSU deployment framework provided by the embodiment of the present disclosure does not rely on manually designed input features.

[0062] The general RSU deployment framework provided by the embodiments of the present disclosure can be deployed to various V2X simulators. V2X simulators refer to various intelligent simulation software related to the Internet of Vehicles, such as SUMO, Carla, etc., which can be used for traffic flow simulation, vehicle interaction behavior simulation, etc. Deployment to V2X simulators means that the deployment scheme obtained by the embodiments of the present disclosure can be used to simulate the deployment of RSU in these simulators to test the performance, and the simulated traffic data generated by these simulators can also be used to provide guidance for the deployment optimization of RSU.

[0063] Combine the following Figures 3 to 5The design of the feature analyzer is described by way of example, but the present disclosure is not limited thereto. Figure 3 It is a schematic diagram of intelligent modeling of RSU deployment provided by an embodiment of the present disclosure.

[0064] A significant advantage of the method provided by the embodiment of the present disclosure is that the input information library 110 and the output indicator library 130 are both adjustable and extensible, that is, the type and quantity of the input data sources contained in the input information library 110 and the type and indicator weights of the deployment performance indicators contained in the output indicator library 130 can be increased, decreased or changed according to actual deployment requirements, which makes it possible to efficiently formulate and update deployment requirements.

[0065] like Figure 3 As shown, the input information library 110 (indicated by J) is an expandable library that can accommodate various input data sources, such as a map 111 of the target area (including a road map in the target area), collected or simulated vehicle trajectories 112 (such as GPS trajectories / GPS data) of vehicles (such as cars), Point of Interest (POI) distribution 113, etc., that is, the input information library 110 can include various input data sources related to the deployment of RSUs.

[0066] The analysis neural network 120 may include a feature extraction network 121. The feature extraction network 121 may be used to extract information from various input data sources (j i ∈J represents the i-th input data source in the input information library J, i is a positive integer greater than or equal to 1 and less than or equal to I, I is the number of input data sources in the input information library J, and I is a positive integer greater than or equal to 1) to extract various basic features to meet the requirements of different angles of RSU deployment, which together provide comprehensive guidance for the deployment of RSU. Different from the method of only considering static roadmaps in the related art, in the embodiment of the present disclosure, a variety of input data sources are considered in the input information library 110, which can be expanded and can incorporate new input data sources, thereby providing more accurate and effective guidance for the deployment of RSU.

[0067] In the embodiment of the present disclosure, the feature extraction network 121 can perform feature extraction on each input data source in a unified output format. For example, for any input data source in the input information library 110, one or more basic features can be extracted, and each basic feature is a feature matrix of a size of 84*84 (this is only used for illustration, but the present disclosure is not limited to this).

[0068] In the disclosed embodiment, the feature extraction network 121 may include one or more feature extraction subnetworks, each feature extraction subnetwork being used to extract corresponding basic features. The network structure of each feature extraction subnetwork depends on the basic features to be extracted. For example, assuming that a certain feature extraction subnetwork is used to extract the basic feature of road density from the map 111, the feature extraction subnetwork includes a convolution layer, and the road density is obtained by convolution processing the road network map image in the map. For another example, assuming that another feature extraction subnetwork is used to extract the basic feature of traffic density from the vehicle trajectory 112, the feature extraction subnetwork can perform data statistical processing on the vehicle trajectory 112 to obtain the traffic density. The disclosed embodiment does not limit the number and network structure of the feature extraction subnetworks.

[0069] Output indicator library 130 (denoted by O, hereinafter referred to as k ∈O represents the kth target deployment performance indicator in the output indicator library O, k is a positive integer greater than or equal to 1 and less than or equal to K, K is the number of target deployment performance indicators, K is a positive integer greater than or equal to 1) can contain various deployment performance indicators, each of which can be used to evaluate the effectiveness of the RSU deployment solution. Different V2X services in different regions / cities may have different requirements for RSU deployment, and therefore require different target deployment performance indicators, such as Figure 3 As shown, for example, it may include any one or more of road coverage, traffic coverage, mobile prediction accuracy, communication quality, vehicle coverage, etc. In the disclosed embodiment, a variety of deployment performance indicators may be combined, and the importance of these deployment performance indicators may be balanced according to the V2X services supported by the RSU (achieved by setting the corresponding indicator weights), and the selected target deployment performance indicators and / or indicator weights may be adaptively adjusted for different target deployment requirements, so as to meet the needs of coexisting and emerging V2X services.

[0070] In the embodiments of the present disclosure, spatial coverage refers to the actual coverage of the RSU, which can be the area actually covered by the signal of the RSU, or the length of the road actually covered by the signal of the RSU, and other specific indicators. Road coverage is a subcategory of spatial coverage, that is, the total length of the road actually covered by the signal of the RSU. Traffic coverage refers to the coverage of the vehicle by the signal emitted by the RSU, which can be the sum of the road traffic flow within the actual coverage of the RSU, or the total number of different vehicles actually covered by the RSU, subject to specific needs. Mobility prediction refers to an application in which the RSU can continuously predict the real-time position of the vehicle after the vehicle leaves the actual coverage of the RSU after collecting the vehicle position information within its actual coverage and training. The mobility prediction accuracy refers to the accuracy of the vehicle position prediction, which is based on a certain position range threshold, that is, when the predicted vehicle position is within a predetermined range of the actual value (the value of the predetermined range can be set according to actual needs, and the present disclosure does not limit this), it is determined to be predicted accurately. Communication quality refers to communication-related indicators such as the speed, congestion level, delay, and jitter of RSU data transmission. Service coverage refers to the duration of V2I (Vehicle to Infrastructure) connection provided by RSU to vehicles within its actual coverage area. Vehicle coverage refers to the number of vehicles actually covered by the signal sent by RSU.

[0071] It is understandable that the deployment performance indicators in the embodiments of the present disclosure are not limited to Figure 3 For example, more deployment performance indicators may be included. For example, some basic deployment performance indicator evaluation mechanisms are given below:

[0072] Road coverage can be calculated by the following formula: Where l(x m ) represents the action x deployed by the mth RSU m The length of the roads covered, m is a positive integer greater than or equal to 1 and less than or equal to M, L represents the total length of all roads in the road network of the target area, M is the total number of RSUs deployed in the target area, and M is an integer greater than or equal to 0.

[0073] Vehicle coverage rate can be calculated by the following formula: Where t(x m ) is the mth RSU deployment action x m The number of vehicles covered, T is the total number of vehicles observed in the road network of the target area, and T is an integer greater than or equal to 1.

[0074] The RSU overlap rate can be calculated by the following formula: Wherein, C is the actual coverage area of ​​all deployed RSUs in the target area, and c is the theoretical coverage area of ​​each RSU.

[0075] like Figure 3 As shown, the analysis neural network 120 in the feature analyzer 100 also includes a fusion network 122. The fusion network 122 can splice the basic features extracted from the feature extraction network 121 that can meet various deployment requirements to generate a basic feature set 1221, and automatically screen and fuse the basic feature set 1221 according to the target deployment performance indicators and their indicator weights obtained from the output indicator library 130, output the fused feature 140, and then input the fused feature 140 into the reinforcement learning network 210.

[0076] The feature analyzer 100 component is critical to achieving good RSU deployment because it determines the quality of the features mined from the input data source. If these features are invalid in the first place, then the subsequent deployment cannot be optimal. In the disclosed embodiment, even for the same set of input data sources, if the target deployment performance indicators obtained from the output indicator library 130 are different, such as the number of extracted target deployment performance indicators is different, and / or the indicator weights of each target deployment performance indicator are different, then the fusion features 140 finally generated are also different, but they are difficult to explicitly model in a traditional manual manner. That is, the basic features extracted from the input data source are instructive for the deployment optimization of a certain target deployment performance indicator, but which basic features are effective for this target deployment task is difficult to determine artificially. For example, assuming that only the deployment optimization is performed for a single target deployment performance indicator, namely road coverage, then the basic feature of road density should play a dominant role in guiding the target deployment task, and the role of other extracted basic features will be weakened or even excluded. In addition, if the target deployment requirement is formed by weighting multiple target deployment performance indicators and their indicator weights, the adjustment of the indicator weight of each target deployment performance indicator will also cause the guiding contribution of different basic features to RSU deployment to change, that is, the importance ranking of each basic feature will change, thereby resulting in different fusion features 140. To this end, the embodiment of the present disclosure designs a novel network structure with two scalable libraries, namely, the input information library 110 and the output indicator library 130, and then proposes an efficient analysis neural network 120, which automatically learns and fuses according to the target deployment requirements to obtain a good feature presentation, such as Figure 3 Fusion feature 140 is shown.

[0077] Figure 4 It is a functional schematic diagram of a fusion network provided by an embodiment of the present disclosure.

[0078] In the disclosed embodiment, the analysis neural network 120 in the feature analyzer 100 is used to complete feature extraction and fusion. In order to facilitate subsequent calculations, the size of all basic features extracted from each type of input data source is unified into a two-dimensional feature frame of the same size (e.g., 84*84). As more two-dimensional feature frames are obtained from the input information library J, more flexibility can be achieved to adjust the later deployment strategy to optimize the target deployment performance indicators. Figure 4 As shown, it is assumed that N basic features are extracted from Z data sources, each of the N basic features is a two-dimensional feature frame of 84*84, and N is a positive integer greater than or equal to Z. It should be noted that the N basic features are not necessarily extracted from N input data sources. Different basic features can be extracted from the same input data source according to different angles, that is, one or more basic features can be extracted from each data source. However, it is impossible to manually determine which basic features to use each time. Therefore, the embodiment of the present disclosure further designs a feature weight adjustment network based on the attention mechanism as a fusion network, which automatically learns the importance of each two-dimensional feature frame according to the required target deployment performance indicators, that is, determines the feature weight of each two-dimensional feature frame.

[0079] Figure 4 In an embodiment, each basic feature is uniformly extracted into the same size according to deployment requirements. For example, the size of all basic features during testing is 84*84. N basic features are concatenated to generate a basic feature set 1221 of 84*84*N size. The basic feature set 1221 of 84*84*N size is processed using a feature weight adjustment network based on an attention mechanism to generate a compressed feature set 1222 of 1*1*N size. A feature weight frame 1223 of 1*1*N size is learned from the compressed feature set 1222. The feature weight frame 1223 includes N feature weights, and each feature weight indicates the importance of the corresponding basic feature in the compressed feature set 1222. The basic feature set 1221 is processed according to the feature weight frame 1223 to obtain the fused feature 140. The fused feature 140 can be subsequently subjected to convolution processing to obtain the fused feature encoding information.

[0080] Specifically, Figure 5 As shown, the fusion network 122 may include a global pooling layer 51, a first fully connected layer 52, a first activation function 53, a second fully connected layer 54, a second activation function 55 and a multiplication component 56. The basic feature set 1221 of size 84*84*N is input to the global pooling layer 51, and a global pooling operation is performed on it to obtain a compressed feature set 1222 of size 1*1*N, and then a multi-layer perceptron (multi-layer perception, MLP) is used to learn the attention weight, that is, the feature weight. Figure 5In an embodiment, the multilayer perceptron may include a first fully connected layer 52, a first activation function 53, and a second fully connected layer 54. A compressed feature set 1222 of size 1*1*N is input to the first fully connected layer 52, and a processed feature set 1224 of size 1*1*Q (Q is adjusted according to actual conditions and is a positive integer less than N) is output. The processed feature set 1224 of size 1*1*Q is processed by the first activation function 53, the second fully connected layer 54, and the second activation function 55 in turn to generate a feature weight frame 1223 of size 1*1*N. In the embodiment of the present disclosure, the first activation function 53 may be, for example, a ReLU activation function, and the second activation function 55 may be, for example, a sigmoid function. In the embodiment of the present disclosure, the sigmoid function is used to obtain normalized weights between 0 and 1 to represent different feature weights. By doing so, the learned attention weight can be regarded as the importance of each basic feature, and then the feature weight is multiplied by the corresponding basic feature through the multiplication component 56 and spliced ​​together to form a fusion feature 140, completing the automatic selection and fusion of the basic features. Different feature weights indicate that the basic feature plays a different role in guiding the RSU deployment task. If the feature weight of a basic feature frame is 0, it means that the basic feature has been screened out. This fusion feature 140 will be used as the key input of the next component, the deployment scheduler 200, to determine the deployment location of each RSU.

[0081] It should be noted that Figure 5 It only shows the network structure of the fusion network. In fact, the training of the fusion network parameters is completed along with the training process of the reinforcement learning network.

[0082] The following describes the design of the deployment scheduler 200. Figure 6 and Figure 7 An example is given, but the present disclosure is not limited thereto. By extracting features useful for RSU deployment, the RSU deployment problem can be accurately described and modeled as a process of determining the best combination of RSU deployment locations to meet the target deployment requirements, as shown below:

[0083]

[0084] stf(J,X)={o k}, where o k ∈O (2)

[0085] |X|≤M, where X∈R (3)

[0086] Among them, k represents the value of the kth target deployment performance indicator obtained under the influence of the corresponding RSU deployment action. k represents the kth target deployment performance indicator ok The kth indicator weight of K is the total number of target deployment performance indicators obtained from the output indicator library O. The goal of this optimization is to maximize the overall RSU deployment utility measured based on K target deployment performance indicators, that is, by optimizing the deployment of each RSU in the target area, maximize the value of the weighted sum of the K target deployment performance indicators and their indicator weights in formula (1). f(J,X) represents the optimization of the kth target deployment performance indicator o given the input data source in the input information library J and the deployment location / deployment site X of each RSU in the target area. k f(J,X)={o k} is used to describe how the input data source and the deployment action of RSU jointly determine the value of each target deployment performance indicator, and f(·) is used to describe this conversion relationship. The constraint |X|≤M is to ensure that the number of RSUs deployed in the target area does not exceed the budget M, and X∈R indicates that the deployment location of RSU should be selected from all feasible locations R on the map of the target area.

[0087] The above optimization problem is the theoretical basis for the design of the embodiment of the present disclosure, but it is not easy to solve this formula directly. Because the function f(·) is difficult to obtain explicitly. Therefore, it is recommended to convert this optimization problem into the following learning-based problem to obtain a practical solution. Figure 6 FIG. 1 is a schematic diagram of an MDP-based RSU deployment process provided by an embodiment of the present disclosure. Figure 6 As shown, deep reinforcement learning conversion is performed. The disclosed embodiment proposes multi-step decision learning to derive a deployment plan, wherein the optimal deployment position in X is generated based on the information available in the input information base J and the order of these already selected deployment positions, that is, the optimal deployment position of the RSU at time t depends on the fused features in the input information base J and the deployment positions of the RSUs that have been deployed from time 0 to time t-1. In addition, multi-step decision learning can be well modeled by MDP, which is a practical framework for solving decision-making problems in uncertain environments by defining a set of states (state, denoted by s below), actions (action, denoted by a below), and rewards (reward, denoted by r below). As Figure 6 As shown, each action deploys an RSU on the road network (contained in the environment 212) of the map of the target area, thereby changing the deployment environment and generating a new state, which is input into the intelligent agent 211 to determine the next RSU deployment site / location, that is, to determine the RSU deployment action at the next moment.

[0088] like Figure 6As shown, the reinforcement learning network 210 may include an agent 211 and an environment 212. The environment 212 for RSU deployment may include a map of the target area, fused features learned from the input data source, and the deployment locations of existing RSUs. The agent 211 may be a neural network, the number of output layers of which is related to the number of candidate locations among all feasible locations R on the map of the target area, that is, the output of the agent 211 may be a two-dimensional distribution, the size of which (assuming 100*100, but this is only used as an example here, and the present disclosure is not limited thereto) is related to the number of candidate locations. The deployment location of the RSU determined by the action is obtained after pruning from the 100*100 matrix.

[0089] MDP is constructed based on a set of interacting objects, namely, agent 211 and environment 212, and its elements include state, action, policy and reward. In the simulation of MDP, agent 211 will perceive the current system state (i.e., state), and take actions on environment 212 according to the policy (i.e., deployment policy), thereby changing the state of environment 212 and obtaining rewards, and the accumulation of rewards over time is called return. In terms of application, MDP is used to model reinforcement learning problems in machine learning. MDP can solve the agent strategy that maximizes the return.

[0090] In order to derive a practical solution to this MDP problem, the DRL framework is adopted in the disclosed embodiment, and the four key elements of the state, action, reward and strategy in the project are designed as follows:

[0091] The state s describes each specific environmental state of the DRL system, which can be represented by the current fused features obtained from the feature analyzer. With this main state information, the encoding position of the deployed RSU and other domain knowledge can be added to improve the stability of the state encoding. Because in reinforcement learning, the more detailed the environmental change information is, the better the training and learning can be. That is, the state s at time t t = fusion features at time t + encoding position of the RSU deployed at time t + other domain knowledge.

[0092] Among them, the encoding position of the RSU deployed at time t can be obtained by Figure 7 The deployed action encoding information 221 in the Figure 7 The deployed action information 220 in the byte is encoded, for example, by using a fully connected layer (FC 72) for encoding, but the present disclosure is not limited thereto. For example, the position of the RSU can be represented by a two-dimensional matrix containing 0 and 1, and the corresponding position is set to 1 if the RSU is deployed, and is set to 0 if the RSU is not deployed. Such a matrix can be easily encoded using a fully connected layer.

[0093] In the embodiment of the present disclosure, other domain knowledge also includes the coverage matrix of currently deployed actions (e.g. Figure 7 The deployed action coverage information 230 in the current deployed action, the mask matrix generated by the current deployed action (e.g. Figure 7 ), these matrices are then used in the same way as above with a fully connected layer (e.g., Figure 7 FC 73 and FC 74) are encoded to obtain the deployed action coverage coding information 231 and the deployed action mask coding information 241 respectively.

[0094] Action a selects a candidate position from R as the next RSU deployment position. For time t, the candidate position needs to exclude the positions of RSUs that have been deployed from time 0 to time t-1. This operation is completed by a mask matrix, see Figure 7 Mask 242 in. In principle, all locations on the road network of the map of the target area can be regarded as potential deployment locations of the RSU, and the deployment granularity can be adjusted according to the computational overhead and the requirements of the V2X service. The deployment granularity is the distance between the candidate deployment locations of the RSU. The deployment granularity here refers to the resolution size of the action space, that is, the size of the candidate action space, which means the adjustable fineness of the action. For example, if the map of the target area is divided into 100*100 granularity, 10,000 candidate action spaces are obtained. If it is divided into 10*10 granularity, 100 candidate action spaces are obtained. Finer-grained divisions can be closer to the actual situation, because they can cover more potential candidate deployment locations, and can also be oriented to more application scenarios, but the corresponding computational complexity will also increase.

[0095] Reward r is the feedback for each deployment action. In the problem of the embodiment of the present disclosure, reward r can be the quantitative impact of the action on the overall deployment index (i.e., the weighted sum of K target deployment performance indicators and their index weights), and reward r is the value used to measure the goodness of a certain action in reinforcement learning. In order to link the reward with each target deployment performance indicator obtained in the output index library O, a series of reward functions are defined. On this basis, the various reward functions can be weighted together by the index weight of the target deployment performance indicator to obtain the reward for a certain action, and maximize the long-term reward to be achieved by the final deployment strategy. This process can be completed through the training process of the reinforcement learning PPO (Proximal Policy Optimization) algorithm. The long-term reward is the sum of the total rewards of all these actions after completing the deployment of all RSUs, which can also be called a cumulative reward.

[0096] In the embodiment of the present disclosure, when designing the actual reward function, one of the target deployment performance indicators can be used as the reward function, or a weighted sum of multiple target deployment performance indicators and their indicator weights can be used as the reward function. The reward design can be designed according to the specific target deployment requirements.

[0097] The strategy π defines the transition probability distribution between states. In the problem of the embodiment of the present disclosure, the function f(·) is the expected strategy for guiding the sequential RSU deployment, that is, the function f(·) is just such a similar strategy, which also needs to be able to provide guidance for the next operation (action) based on the current information (state, reward). Through repeated events (consisting of a sequence of states, actions, and rewards), a strategy π is trained using the neural network in the agent 211. θ , learning to take the best action for a specific state. The trained π is the best deployment strategy under ideal conditions. The policy parameters θ can be trained using the PPO algorithm, which uses the method of clipping the objective function to improve the training efficiency of reinforcement learning. The policy refers to the state s at the current time t t and reward r t Under the premise of determining the RSU deployment position a at the next time t+1 t+1 The conditional probability can be calculated based on the current state s t Give the next action a t+1 The probability distribution of .

[0098] Assume that the event at time t = the state s at time t t +Action a at time t t+1 +t time reward r t . Figure 6 The neural network in agent 21 takes the event at each moment as a training sample, such as the state s at moment t. t and the reward r at time t t As the input data of agent 211, the neural network predicts an action a' at time t+1 t+1 , and then according to the predicted action a' t+1 and real action t+1 Train the neural network and determine the parameters θ of the neural network, thereby determining the strategy π θ , that is, strategy π θ is a neural network with the parameter θ. After the training is completed, the state and reward at a certain moment are input into the neural network, and the neural network can automatically output the best action at the next moment, that is, determine the best deployment position of the RSU at the next moment.

[0099] Figure 7The network structure of the DRL-based RSU deployment framework provided by the embodiment of the present disclosure is shown, which includes a policy network 700 and a value network 800. The inputs of the policy network 700 and the value network 800 are both state embeddings 77, which are obtained by concatenating feature embeddings obtained by processing various types of information observed in the reinforcement learning environment through fully connected layers (FC). Specifically, the fused feature 140 is processed by FC71 to obtain the fused feature coding information 141, the deployed action information 220 is processed by FC 72 to obtain the deployed action coding information 221, the deployed action coverage information 230 is processed by FC 73 to obtain the deployed action coverage coding information 231, and the deployed action mask information 240 is processed by FC 74 to obtain the deployed action mask coding information 241. The fused feature coding information 141, the deployed action coding information 221, the deployed action coverage coding information 231 and the deployed action mask coding information 241 are concatenated to obtain the state embedding 77, that is, the state is represented by the state embedding 77 after encoding.

[0100] Figure 7 In the embodiment, the policy network 700 and the value network 800 are an actor-critic network, which is a reinforcement learning network that combines policy gradient and temporal difference learning. The actor refers to the policy function π θ , that is, learning a strategy to get the highest possible reward. The critic refers to the value function, which estimates the value function of the current strategy, that is, evaluating the actor's performance.

[0101] Specifically, the policy network 700 includes a fully connected layer ( Figure 7 FC 75 in ) and a deconvolution network structure 710. The state embedding 77 is first input to FC 75 to convert the state embedding 77 into a format suitable for processing by the deconvolution network structure 710. The deconvolution network structure 710 can generate a two-dimensional probability distribution in the action space. The value network 800 may include an MLP network 76, which may include two hidden layers (e.g., 512*512, but only for illustration here) for predicting the expected reward estimate of the current position. The policy network 700 is then trained and optimized to maximize the expected reward estimated by the value network 800. Figure 7 In the example, the reward input from the environment to the agent is the reward for a single-step action determined by the designed reward function. The policy network 700 is used to predict the long-term reward of all actions when deploying the target area, that is, the sum of the rewards of all actions. The training optimization of the policy network 700 is to maximize the long-term reward predicted by the value network 800.

[0102] Figure 7 In the embodiment, in order to improve learning efficiency, a mask 242 is also designed. For example, the deployed action mask information 240 can be used as the mask 242, and the mask 242 is multiplied by the output of the deconvolution network structure 710 to output the masked strategy 716. The masked strategy 716 predicts the next action, and the environment is changed by the predicted next action to generate new states and rewards, so as to filter out unnecessary or unfeasible RSU deployment locations / positions before each action sampling, thereby improving deployment efficiency and accuracy. The embodiment of the present disclosure can speed up the training process by adding a mask 242 to filter and limit the output of the policy network 700. In addition, Figure 7 In the figure, the fusion network 122 and the reinforcement learning network 210 are linked together. During training, the parameters of the fusion network 122 used to determine the feature weights are learned and updated together with the parameters of the reinforcement learning network 210. Therefore, the feature weights determined by the fusion network can reflect the selection of the target deployment performance indicators and the setting of their indicator weights.

[0103] In the disclosed embodiment, the fusing network 122, the policy network 700 and the value network 800 can be jointly trained by the following algorithm, so that the network parameters of the fusing network 122 are trained together with the parameters of the policy network 700, that is, the fusing network 122 is connected to the reinforcement learning network, so during the training process, the rewards collected by the reinforcement learning network (that is, the actual output value of the target deployment performance indicator on each action weighted by the indicator weight) can be fed back to all network parameter updates, including the network parameters of the fusion network and the parameters of the policy network.

[0104] In the following training algorithm, a represents action, v represents value, s represents state, r represents reward, env represents RL environment, and TrajEnd() indicates that all RSUs have been deployed.

[0105] The input of the algorithm includes fused features extracted from the fusion network;

[0106] Initialize: fusing network and policy networkθ, value networkθ ν ;

[0107] / / Initialize the parameters θ of the fusion network and the policy network, and the parameters θ of the value network Vν .

[0108] For epoch=1,2,…,max_epochs do

[0109] / / Based on the current strategy π θ Generate some RSU deployment strategies. Each cycle in this big cycle (epoch) means that under the current network parameters, the agent interacts with the environment in a series of ways, and a batch of data is collected for parameter update. max_epochs refers to the number of big cycles, which can be set according to actual needs and is a positive integer greater than or equal to 1.

[0110] buffer.clear(); / / Clear the buffer before each large loop execution

[0111] for i=1,2,…,steps_per_epoch do

[0112] / / This inner loop (i) determines how many deployment plan data need to be collected by setting steps_per_epoch (which can be set according to actual needs and is a positive integer greater than or equal to 1). For example, if a deployment plan is to deploy 32 RSUs, then if this value is set to 128, it means that the agent needs to interact with the environment to generate 4 complete deployment plans. Here, a complete deployment plan is called a Trajectory, abbreviated as Traj.

[0113] log p←π(a|θ); / / The current strategy π(a|θ) generates a probability distribution log p, which is the last square 711 in the deconvolution network structure.

[0114] log p←Mask(log p) / / Use mask 242 to perform action filtering on the current probability distribution log p to obtain the masked strategy 716, i.e., “log p” on the left side here.

[0115] a←log p.sample(); / / Sampling actions from this probability distribution (masked strategy 716).

[0116] v←V(θ v ); / / By the current value network V(θ v ) generates a value assessment v.

[0117] s,r←env.step(a); / / The state s generated by the action a and the reward r obtained are given by the environment env.

[0118] buffer.append(log p,a,v,s,r); / / Store the above data log p,a,v,s,r into buffer buffer.

[0119] if TrajEnd()then

[0120] env.reset(); / / Environment reset

[0121] end if

[0122] end for

[0123] / / Update the policy by maximizing the PPO-Clip objective (4) and using the commonly used Adam for gradient descent calculations.

[0124] dθ←compute_loss_pi(buffer); / / This means calculating the gradient loss of the current policy network based on the currently collected data (data cached in the buffer). Specifically, according to V(θ v ) calculates the advantage estimate, calculates the ratio of the new and old strategies based on the new and old log p, and calculates the gradient loss dθ based on the objective (4).

[0125] updateθ; / / Gradient descent to update parameters.

[0126] dθ v ←compute_loss_v(buffer); / / This means calculating the gradient loss of the current value network based on the currently collected data. Specifically, the mean square error is calculated based on the actual reward and the output estimate of the value network to obtain

[0127] updateθ v ; / / Gradient descent to update parameters.

[0128] end for

[0129] PPO-Clip Target L CLIP (θ) can be calculated by the following formula:

[0130]

[0131] This formula (4) is the objective function optimized by the PPO algorithm, which can be used for the calculation of updating the above parameter θ. Where ε represents the shear rate.

[0132] In the above formula, is the expected reward of parameter θ at time step t, r t represents the ratio of the new strategy to the old strategy, To estimate the advantage. Referring to the training algorithm above, it can be seen that the reward is collected together with other variables such as state, action and other information and then used to update the network parameters. The reward is related to the target deployment performance index and its index weight. Therefore, the feature weight determined by the fusion network during the training process changes with the change of the target deployment performance index and its index weight. It can be understood that during the training process, as the network parameters of the fusion network are updated, the feature weight determined in each time step t also changes accordingly. Correspondingly, the fusion features determined at each time step t also change.

[0133] It should be noted that the network structures of the fusion network, policy network and value network in the embodiments of the present disclosure are not limited to the above examples. For example, the deconvolution network structure of the policy network can also be replaced by MLP.

[0134] Figure 8 Schematic diagram of an action space initialization filtering operation provided by an embodiment of the present disclosure. Figure 8 As shown, the map of the area to be deployed (i.e., the target area) is set up with a road buffer using geographic information processing software such as QGIS, thereby obtaining a bold road network 80, and then the bold road network 80 is used to filter the original action space (such as an action space of 100*100 size) matrix, thereby excluding candidate positions outside the road network, thereby reducing the amount of data processing and improving the deployment accuracy of the RSU. The original action space matrix refers to the feasible candidate positions for RSU deployment determined according to the set deployment granularity before the RSU deployment task is performed in the area to be deployed. For example, if the area to be deployed is divided into a two-dimensional matrix of 100*100, the original action space matrix has 10,000 candidate positions. Figure 8 81 in the figure represents the action space matrix of the candidate positions after excluding the road network, 811 is a local part of the action space matrix 81, an enlarged schematic diagram of the local part 811 is shown on the right, and 8111 represents the deployment position of an RSU in the local part 811.

[0135] In the disclosed embodiment, after each deployment action, the action space can be filtered again to exclude those inefficient actions. Figure 7As shown, the deconvolution network structure 710 is multiplied by the mask 242. Since the white area corresponding to the deployed action mask information 240 in the mask 242 can be set to 0 and the black area can be set to 1, the white area belongs to the theoretical coverage range of the signal of the deployed RSU, so deploying RSU in the white area is an inefficient action, and these inefficient actions can be removed by the mask 242. That is, the filter mask enhanced by other domain knowledge can significantly reduce the action space, so that the reinforcement learning algorithm can explore more efficiently.

[0136] After the above elements are designed, they are further organized into a well-structured algorithm framework to obtain the ideal RSU deployment strategy, which can include the following steps:

[0137] Step 1: Determine the area to be deployed, the total number of RSUs to be deployed in the area to be deployed, and the deployment locations of the RSUs deployed in the area to be deployed, and load the input data source from the input information base J.

[0138] Step 2: Select the required target deployment performance indicators from the output indicator library O, determine the numerical requirements and indicator weights of each target deployment performance indicator, and load or design the corresponding reward function based on the determined target deployment performance indicators and their indicator weights.

[0139] Step 3: Execute the PPO-based algorithm proposed in the embodiment of the present disclosure to train the above-mentioned DRL framework.

[0140] Step 4: After training convergence, the framework outputs a deployment solution containing all RSU locations.

[0141] The disclosed embodiment proposes a general RSU deployment modeling and planning method, which can be applied to large-scale deployment of roadside infrastructure, can handle various complex deployment scenarios, and provide in-depth guidance for the deployment of RSU. Different from the existing heuristic deployment scheme, the disclosed embodiment innovatively proposes to model the RSU deployment problem in a data-driven form, and deploy the RSU site in a more scientific way. To this end, on the one hand, the disclosed embodiment develops a feature analyzer to extract the basic features of various types of input data sources, and fuses the basic features of different feature weights according to the deployment goals, so that the basic features for guiding RSU deployment do not need to be manually determined, but can be automatically learned by the feature analyzer. At the same time, the input data source and the output target deployment performance index of the proposed RSU deployment framework can be extended to suit various deployment requirements. By doing so, the method can be compatible with all deployment requirements with a small number of modifications. On the other hand, considering the long-term incremental nature of RSU deployment, MDP is introduced to provide optimization for the RSU deployment planning process. Specifically, the disclosed embodiment customizes a reinforcement learning model to intelligently explore the best deployment strategy. This is superior to related technologies that use heuristic algorithms or intelligent optimization algorithms to give a one-time deployment solution. The proposed deployment strategy based on reinforcement learning can give the deployment action relative to the global optimality according to the actual situation at each moment, rather than being limited to a fixed deployment scheme. The disclosed embodiment integrates this entire set of deployment and planning solutions into a complete framework that is highly scalable and compatible, and reserves a rich interface for its interface with the V2X simulator.

[0142] The disclosed embodiments propose a general RSU deployment framework to improve related technologies through better utility design and deployment scalability. The RSU deployment framework provided by the disclosed embodiments integrates an input information library and an output indicator library, both of which are adjustable and scalable to take into account rich sensor data or emerging / upgraded deployment requirements related to RSU deployment. A feature analyzer is also designed to automatically learn the complex relationship between the basic features extracted from the input data source and the target deployment requirements obtained from the output indicator library. In the policy training phase, reinforcement learning (RL) is used to model the general RSU deployment problem as a learning process, and the RL model is customized to automatically explore the deployment environment to find a good deployment strategy. The solution provided by the disclosed embodiments can be applied to large-scale deployment scenarios of RSU, even if the deployment requirements are complex, diverse and changeable, and it is also more efficient in exploring deployment solutions.

[0143] With the foreseeable large-scale deployment of C (Cellular)-V2X, the deployment modeling and planning methods proposed in the embodiments of the present disclosure will bring great value to the industry. For example, for highways, the system can customize deployment plans based on the perception needs of vehicles entering the highways and support advanced new data-driven C-V2X use cases; for urban roads, various perception facilities can be integrated and further deployed in combination with actual deployment needs to provide value for urban management; for airports, smart campuses, ports, etc., infrastructure and related road users are generating more and more data, which will help the commercialization of the project and realize new use cases.

[0144] Fig. 9 1 is a schematic diagram of the structure of a communication device deployment apparatus provided by an embodiment of the present disclosure. The communication device deployment apparatus may be a computer program (including program code, etc.) running on a computer device, for example, the communication device deployment apparatus may be an application software; the apparatus may be used to execute the corresponding steps of the method provided by an embodiment of the present disclosure. Fig. 9 As shown, the communication device deployment apparatus 900 can be used for the computer device in the embodiment corresponding to the above figure. Specifically, the apparatus may include: a determination unit 910 and a processing unit 920.

[0145] The determination unit 910 is used to determine a target deployment performance indicator that meets the target deployment requirements.

[0146] The processing unit 920 is used to process the input data source based on the target deployment performance indicator by analyzing the neural network, so as to extract fusion features that meet the target deployment performance indicator from the input data source.

[0147] The processing unit 920 is also used to process the fusion feature through a reinforcement learning network to determine a target deployment plan of the communication device in a target area, so as to maximize the cumulative reward of the reinforcement learning network determined based on the target deployment performance indicator.

[0148] In an exemplary embodiment, the determination unit 910 is further used to: provide an expandable output indicator library, the output indicator library including various types of deployment performance indicators; and select a target deployment performance indicator that can meet the target deployment requirements from the deployment performance indicators in the output indicator library.

[0149] In an exemplary embodiment, the communication equipment deployment apparatus 900 further includes: a setting unit, configured to set an indicator weight of each target deployment performance indicator according to the target deployment requirement.

[0150] Among them, the processing unit 920 is also used to: process the input data source based on the target deployment performance indicator and its indicator weight through the analysis neural network, so as to extract the fusion features that meet the target deployment performance indicator and its indicator weight from the input data source.

[0151] In an exemplary embodiment, the communication device deployment apparatus 900 further includes: a providing unit, configured to provide an expandable input information library, wherein the input information library includes various types of input data sources.

[0152] In an exemplary embodiment, the analysis neural network includes a feature extraction network and a fusion network.

[0153] Among them, the processing unit 920 is also used to: input Z input data sources into the feature extraction network to obtain N basic features of the Z input data sources, Z is a positive integer greater than or equal to 1, and N is a positive integer greater than or equal to Z; splice the N basic features of the Z input data sources to generate a basic feature set; process the basic feature set through the fusion network to generate a compressed feature set, and learn a feature weight frame based on the compressed feature set, wherein the feature weight frame includes N feature weights, and each feature weight indicates the importance of the corresponding basic feature; based on the feature weight frame and the basic feature set, generate the fused feature.

[0154] In an exemplary embodiment, the determination unit 910 is further configured to: determine the rewards collected by the reinforcement learning network according to the target deployment performance indicator. The processing unit 920 is further configured to: jointly train the network parameters of the fusion network and the parameters of the reinforcement learning network based on the collected rewards, so that the feature weight frame determined based on the fusion network is associated with the target deployment performance indicator.

[0155] In an exemplary embodiment, the reinforcement learning network includes an environment and an agent.

[0156] Among them, the processing unit 920 is also used to: change the environment based on the action output by the agent, the action is used to deploy the communication device in the target area; determine the state and reward observed from the environment based on the changed environment, the state observed from the environment includes the fusion feature and the deployed action information of the target area, and the reward is determined according to the target deployment performance indicator; process the state and the reward through the agent, and output the next action, and the cumulative reward is the accumulation of rewards generated by each action.

[0157] In an exemplary embodiment, the state observed from the environment further includes deployed action coverage information and / or deployed action mask information.

[0158] In an exemplary embodiment, the state observed from the environment further includes deployed action mask information.

[0159] Among them, the processing unit 920 is also used to: generate a mask based on the deployed action mask information, wherein the mask indicates the signal coverage of the communication equipment deployed in the target area; and use the mask to remove the signal coverage of the communication equipment deployed in the target area before outputting the next action through the agent based on the state and the reward.

[0160] In an exemplary embodiment, the communication device deployment apparatus 900 further includes: an acquisition unit, configured to acquire a map of the target area. The processing unit 920 is further configured to: process the map to determine a candidate communication device deployment area in the map; and filter the actions that the agent can output based on the candidate communication device deployment area.

[0161] An embodiment of the present disclosure provides a communication device deployment apparatus, which can be run in a computer device.

[0162] Fig.10 FIG. 1 is a schematic diagram of an operating environment in some embodiments of the present disclosure. Fig.10 As shown, the terminal devices of multiple users (eg, including user devices 101a, 101b, and 101c) are connected to the server 102 via the network 106. The server 102 maintains a database 104, which stores network data for providing data to the user devices.

[0163] The user device may provide the target deployment requirements to the server 102, and the server 102 may determine the target deployment performance index based on the target deployment requirements received from the user device, and process the input data source based on the target deployment performance index by analyzing the neural network to extract the fusion features that meet the target deployment performance index from the input data source; the fusion features are processed by the reinforcement learning network to determine the target deployment scheme of the communication device in the target area, so as to maximize the cumulative reward of the reinforcement learning network determined based on the target deployment performance index. The server 102 may provide the target deployment scheme to the user device. The user device displays based on the received target deployment scheme.

[0164] The user device may be a mobile phone (such as user device 101c) or a laptop (such as user device 101b), or a playback device (such as user device 101a), etc., without limitation. Fig.10 The user equipment in the example is only a part of the equipment, and the user equipment in the present disclosure is not limited to the Fig.10 The equipment listed in .

[0165] It is understandable that the user device mentioned in the embodiment of the present disclosure may be a computer device, and the computer device in the embodiment of the present disclosure includes but is not limited to a terminal device or a server. In other words, the computer device may be a server or a terminal device, or a system composed of a server and a terminal device. Among them, the terminal device mentioned above may be an electronic device, including but not limited to a mobile phone, a tablet computer, a desktop computer, a laptop computer, a PDA, a vehicle-mounted device, an augmented reality / virtual reality (AR / VR) device, a helmet display, a smart TV, a wearable device, a smart speaker, a digital camera, a camera, and other mobile Internet devices (mobile internet device, MID) with network access capabilities, or terminal devices in scenarios such as trains, ships, and flights. Among them, the server mentioned above may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, vehicle-road collaboration, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms.

[0166] Optionally, the data involved in the embodiments of the present disclosure may be stored in a computer device, or the data may be stored based on cloud storage technology, which is not limited here.

[0167] See also Fig.11 , Fig.11 Schematic diagram of a computer device provided by an embodiment of the present disclosure. Fig.11 As shown, the computer device in the embodiment of the present disclosure may include: one or more processors 111, a memory 112, and an input-output interface 113. The processor 111, the memory 112, and the input-output interface 113 are connected via a bus 114. The memory 112 is used to store a computer program, which includes program instructions. The input-output interface 113 is used to receive data and output data, such as for data interaction between a host machine and a computer device, or for data interaction between various virtual machines in a host machine; the processor 111 is used to execute the program instructions stored in the memory 112.

[0168] Among them, the processor 111 can perform the following operations: determine the target deployment performance indicator that meets the target deployment requirements; process the input data source based on the target deployment performance indicator by analyzing the neural network to extract fusion features that meet the target deployment performance indicator from the input data source; process the fusion features through the reinforcement learning network to determine the target deployment plan of the communication device in the target area, so as to maximize the cumulative reward of the reinforcement learning network determined based on the target deployment performance indicator.

[0169] In some feasible implementations, the processor 111 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0170] The memory 112 may include a read-only memory and a random access memory, and provide instructions and data to the processor 111 and the input / output interface 113. A portion of the memory 112 may also include a non-volatile random access memory. For example, the memory 112 may also store information about the device type.

[0171] In a specific implementation, the computer device can execute the implementation methods provided in the steps in the above figure through its built-in functional modules. For details, please refer to the implementation methods provided in the steps in the above figure, which will not be repeated here.

[0172] The embodiment of the present disclosure provides a computer device, including: a processor, an input and output interface, and a memory. The processor obtains a computer program in the memory to execute each step of the method shown in the above figure.

[0173] The embodiment of the present disclosure also provides a computer-readable storage medium, which stores a computer program, and the computer program is suitable for being loaded by the processor and executing the communication equipment deployment method provided by each step in the above figure. For details, please refer to the implementation method provided by each step in the above figure, which will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in the present disclosure, please refer to the description of the method embodiment of the present disclosure. As an example, the computer program can be deployed to be executed on one computer device, or on multiple computer devices located in one place, or on multiple computer devices distributed in multiple locations and interconnected by a communication network.

[0174] The computer-readable storage medium may be the communication device deployment device provided in any of the aforementioned embodiments or the internal storage unit of the computer device, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (smart media card, SMC), a secure digital (secure digital, SD) card, a flash card (flash card), etc. equipped on the computer device. Further, the computer-readable storage medium may also include both the internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0175] The embodiment of the present disclosure also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in the various optional modes in the above figure.

[0176] The terms "first", "second", etc. in the specification, claims, and drawings of the embodiments of the present disclosure are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, product, or equipment that includes a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules that are not listed, or may optionally include other step units inherent to these processes, methods, devices, products, or equipment.

[0177] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in this description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0178] The method and related apparatus provided by the embodiments of the present disclosure are described with reference to the method flow chart and / or structural diagram provided by the embodiments of the present disclosure. Specifically, each process and / or block in the method flow chart and / or structural diagram, as well as the combination of the processes and / or blocks in the flow chart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable search device to generate a machine, so that the instructions executed by the processor of the computer or other programmable search device generate instructions for implementing the process in the process. Figure 1 A process or multiple processes and / or structures Figure 1 These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable search device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including the instruction device, which implements the function specified in the process. Figure 1 A process or multiple processes and / or structures Figure 1 These computer program instructions can also be loaded onto a computer or other programmable search device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A flow or multiple flows and / or structures illustrate the steps of the functions specified in one block or multiple blocks.

[0179] The above disclosure is only a preferred embodiment of the present disclosure, which certainly cannot be used to limit the scope of rights of the present disclosure. Therefore, equivalent changes made according to the claims of the present disclosure are still within the scope of the present disclosure.

Claims

1. A communication equipment deployment method, characterized in that: include: Determine target deployment performance indicators that meet target deployment requirements; Processing the input data source based on the target deployment performance indicator by analyzing the neural network to extract fusion features that meet the target deployment performance indicator from the input data source; The fusion features are processed through a reinforcement learning network to determine a target deployment scheme of the communication device in a target area, so as to maximize the cumulative reward of the reinforcement learning network determined based on the target deployment performance indicator.

2. The method according to claim 1, characterized in that Determine target deployment performance indicators that meet target deployment requirements, including: Providing an extensible output indicator library, the output indicator library includes various types of deployment performance indicators; A target deployment performance indicator that can meet the target deployment requirement is selected from the deployment performance indicators in the output indicator library.

3. The method according to claim 2, characterized in that Also includes: Setting the indicator weight of each target deployment performance indicator according to the target deployment requirements; The step of processing the input data source based on the target deployment performance indicator by analyzing the neural network to extract fusion features that meet the target deployment performance indicator from the input data source includes: The input data source is processed by the analysis neural network based on the target deployment performance indicator and its indicator weight, so as to extract the fusion feature that meets the target deployment performance indicator and its indicator weight from the input data source.

4. The method according to any one of claims 1 to 3, characterized in that Also includes: An expandable input information library is provided, wherein the input information library includes various types of input data sources.

5. The method according to claim 1, characterized in that The analytical neural network includes a feature extraction network and a fusion network; The step of processing the input data source based on the target deployment performance indicator by analyzing the neural network to extract fusion features that meet the target deployment performance indicator from the input data source includes: Inputting Z input data sources into the feature extraction network to obtain N basic features of the Z input data sources, where Z is a positive integer greater than or equal to 1, and N is a positive integer greater than or equal to Z; Concatenate the N basic features of the Z input data sources to generate a basic feature set; Processing the basic feature set through the fusion network to generate a compressed feature set, and learning a feature weight frame based on the compressed feature set, wherein the feature weight frame includes N feature weights, each feature weight indicating the importance of a corresponding basic feature; The fused feature is generated based on the feature weight frame and the basic feature set.

6. The method according to claim 5, characterized in that Also includes: determining a reward collected by the reinforcement learning network according to the target deployment performance indicator; The network parameters of the fusion network and the parameters of the reinforcement learning network are jointly trained based on the collected rewards so that the feature weight frame determined based on the fusion network is associated with the target deployment performance indicator.

7. The method according to claim 1, characterized in that The reinforcement learning network includes an environment and an agent; The step of processing the fusion features through a reinforcement learning network to determine a target deployment scheme of the communication device includes: changing the environment based on an action output by the agent, the action being used to deploy the communication device in the target area; Determining a state and a reward observed from the environment based on the changed environment, wherein the state observed from the environment includes the fused feature and deployed action information of the target area, and the reward is determined according to the target deployment performance indicator; The state and the reward are processed by the agent to output the next action, and the cumulative reward is the accumulation of rewards generated by various actions.

8. The method according to claim 7, characterized in that The state observed from the environment also includes deployed action coverage information and / or deployed action mask information.

9. The method according to claim 7, characterized in that The state observed from the environment also includes deployed action mask information; The agent processes the state and the reward and outputs the next action, including: generating a mask according to the deployed action mask information, wherein the mask indicates a signal coverage range of the communication device deployed in the target area; Before the agent outputs a next action based on the state and the reward, the mask is used to remove the signal coverage of the communication device deployed in the target area.

10. The method according to any one of claims 7 to 9, characterized in that: Also includes: Obtaining a map of the target area; Processing the map to determine candidate deployment areas of communication devices in the map; The actions that the agent can output are filtered based on the candidate deployment area of ​​the communication device.

11. A communication equipment deployment device, characterized in that: include: A determination unit, used to determine a target deployment performance indicator that meets a target deployment requirement; A processing unit, configured to process an input data source based on the target deployment performance indicator by analyzing a neural network, so as to extract a fusion feature satisfying the target deployment performance indicator from the input data source; The processing unit is further used to process the fusion feature through a reinforcement learning network to determine a target deployment scheme of the communication device in a target area, so as to maximize the cumulative reward of the reinforcement learning network determined based on the target deployment performance indicator.

12. A computer device, characterized in that: Includes processor, memory, input and output interfaces; The processor is connected to the memory and the input / output interface respectively, wherein the input / output interface is used to receive and output data, the memory is used to store a computer program, and the processor is used to call the computer program so that the computer device executes the method described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded and executed by a processor, so that a computer device having the processor executes the method according to any one of claims 1 to 10.

Images