Multi-agent visual data unloading method based on information perception and reinforcement learning

Abstract

The invention belongs to the technical field of data processing, and discloses a multi-agent visual data unloading method based on information perception and reinforcement learning, and the method comprises the following steps: collecting and processing vehicle state and task data based on an edge computing device, and forming training set data; training a multi-agent reinforcement learning network based on the DQN, and constructing a reinforcement learning decision model, namely a DQN model; and the edge computing device distributes the trained DQN model to each vehicle entering the communication range of the edge computing device, and the vehicle makes an autonomous decision to determine the compression ratio of the picture sent to the edge computing device. According to the multi-agent visual data unloading method based on information perception and reinforcement learning, on the basis of the DQN, each vehicle can flexibly adjust the compression level of the image according to the real-time calculation resource load, and the method aims at balancing the time consumption and the processing precision of visual data processing in the resource-limited vehicle environment.

Description

Technical Field

[0001] The present invention relates to the technical field of reinforcement learning, and in particular to a multi-agent visual data unloading method based on information perception and reinforcement learning. Background Art

[0002] In recent years, with the rapid development of Internet technology, intelligent driving technology has gradually matured. At present, intelligent vehicles mainly rely on advanced hardware sensor systems and artificial intelligence algorithms to perceive and analyze road conditions in real time. Visual data has become an important part of sensor data because of its rich details, which can provide direct insights into road conditions. However, high-end models are difficult to be widely used in the short term due to their high costs. In addition, visual data processing requires a lot of computing resources, especially in low-cost vehicles, whose on-board computing power often cannot meet the demand. To solve this problem, vehicle-mounted fog computing (VFC) has gradually become a viable solution. VFC transforms public transportation vehicles or roadside units with sufficient space and power supply into fog nodes, enabling customer vehicles to transmit visual data to fog nodes through vehicle-to-vehicle (V2V) communication to utilize abundant edge computing resources for data processing.

[0003] Although VFC shows potential in visual data processing, it still faces several challenges. First, the amount of visual data is huge and often redundant, and directly transmitting all the raw data will exhaust limited resources. Second, although pre-processing techniques such as compression and resolution adjustment can reduce the amount of data, improper compression methods may significantly reduce recognition accuracy. Therefore, the trade-off between time consumption and processing accuracy becomes a major problem. In addition, the complexity of the vehicle environment adds to the challenges, and the fluctuations in the number of fog nodes and customer vehicles cause instability in service supply and demand.

[0004] In summary, existing technologies have obvious defects in visual data processing, resource utilization and service quality assurance, and improved solutions are urgently needed to enhance the overall performance of intelligent driving systems. Summary of the invention

[0005] The purpose of the present invention is to provide a multi-agent visual data offloading method based on information perception and reinforcement learning, which is based on DQN reinforcement learning to determine how visual data is compressed before offloading, aiming to balance the time consumption and processing accuracy of visual data processing in a resource-constrained VFC environment.

[0006] To achieve the above object, the present invention provides a multi-agent visual data unloading method based on information perception and reinforcement learning, comprising the following steps: Step S1: Collect vehicle status and task data based on edge computing devices, process the data set, and construct training set data; Step S2: training a multi-agent reinforcement learning network based on a deep Q network to construct a DQN model for decision making; Step S3: After the deep Q network training is completed, the edge computing device distributes the trained DQN model to each vehicle that enters its communication range. After the vehicle obtains the trained deep Q network, it makes decisions independently to determine the compression ratio of the image sent to the edge computing device.

[0007] Preferably, in step S1, the specific process of training set data collection and processing is as follows: Step S11: Each vehicle that enters the communication range of the edge computing device is regarded as an intelligent entity. The vehicle first obtains a visual image and records the time when the image is obtained; based on the edge computing device, the relevant information of the image processing task generated by the vehicle connected to it at the current moment is obtained; Step S12: After completing the information collection, the edge computing device processes each image with different compression ratios. The specific compression ratios are: 1.0, 0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2 and 0.1. Step S13: The edge computing device generates the state required for deep Q network training based on the image information sent by different intelligent agents, and records the accuracy of each image after being processed by a specific application and the total time taken for the image to be generated and processed, so as to facilitate subsequent training of the DQN in the edge computing device.

[0008] Preferably, in step S11, the relevant information of the image processing task generated by the vehicle at the current moment includes: two-dimensional information entropy of the image, the original image file, and the number of vehicles that the edge computing device can connect to at the same time; Among them, the two-dimensional information entropy of the image is used to evaluate the complexity and information content of the image; the original image file is used for subsequent processing.

[0009] Preferably, in step S12, for each compression ratio, the edge computing device respectively records the information entropy of each image, different compression ratios and corresponding compression time, the processing time of the YOLO model, and the image recall rate after the YOLO model processing; Among them, the information entropy of each image is used to analyze the impact of different compressions on image information; the processing time of the YOLO model is used to evaluate the processing speed of the model under different image qualities; the image recall rate after processing by the YOLO model is used to evaluate the recognition accuracy of the image after being processed by the YOLO model.

[0010] Preferably, in step S2, the training process of the multi-agent reinforcement learning network based on DQN is as follows: Step S21, first, define the state state as a triple (e, s, F); where e represents the information entropy of the image, reflecting the complexity and information content of the image; s represents the original size of the image, representing the visual data size of the visual data captured by the vehicle set; F represents the computing resource load on the edge computing node, representing the number of vehicles simultaneously connected to the edge computing device at the current moment; Step S22: define the action space action, as shown below: ; Among them, 1, 0.9, 0.8, ..., 0.1 represent the image compression ratio; Step S23: With the goal of minimizing the time consumption T and maximizing the recognition accuracy G of all images, define the reward of reinforcement learning as follows: ; in, Represents the average recognition accuracy of all images processed by the edge computing node. Represents the average total processing time of all images processed by the edge computing node, that is, the sum of compression time and model processing time; is a weight factor used to adjust the DQN model and the degree of emphasis; Step S24, therefore, the final optimization goal of DQN reinforcement learning is as follows: .

[0011] Preferably, in step S3, the trained DQN model is distributed to each vehicle that enters its communication range, and the specific process of the vehicle making autonomous decisions is as follows: Step S31: When the vehicle has a new visual data processing task, the computing device on the vehicle calculates the information entropy of the image and the original size of the image, and the vehicle sends information to the edge computing device to learn the total number of connected vehicles; Step S32: Then, the information entropy of the image, the original size of the image, and the total number of vehicles connected to the edge computing device are used as states and input into the trained DQN model. Then, a corresponding decision is made based on the states to give the optimal compression ratio for the image. Step S33: Finally, the vehicle adjusts the resolution of the image according to the optimal compression ratio given by the DQN model, and then sends the image with adjusted resolution to the edge computing device for subsequent processing.

[0012] Preferably, since the traffic volume in different areas at different times may be significantly different, when the traffic volume changes at different times of the day, the deep Q networks in the edge computing devices in different areas can achieve mutual transmission.

[0013] Preferably, the edge computing device trains the deep Q network model under different traffic flow pressures for different traffic flows, and switches the deep Q network model assigned to the vehicle when the traffic flow changes, thereby ensuring the efficiency of the compression scheme.

[0014] Therefore, the present invention adopts the above-mentioned multi-agent visual data unloading method based on information perception and reinforcement learning, and introduces image information entropy, image original size and the number of vehicles simultaneously connected to the edge device as decision-making basis, so that each vehicle can flexibly adjust the compression level of the image according to the real-time computing resource load, significantly reducing the transmission cost of redundant visual data and improving the overall efficiency of visual data processing; adopts experience-based multi-agent reinforcement learning (MARL) to give each vehicle the ability to make autonomous decisions, so that it can flexibly respond to complex traffic conditions in a fully observable environment, without relying on traditional complex mathematical modeling, but learning through historical experience, automatically adjusting the unloading strategy, so that the system can quickly adapt to dynamically changing traffic environments; through simulation evaluation on real-world traffic data, the significant advantages of the present invention in visual data unloading efficiency and recognition accuracy have been verified, which not only improves the performance of existing intelligent driving systems, but also opens a new market door for intelligent driving applications of low-cost vehicles.

[0015] The technical solution of the present invention is further described in detail below through the drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of the multi-agent visual data unloading method based on information perception and reinforcement learning of the present invention; Figure 2 It is a distributed framework diagram of multi-agent reinforcement learning of the present invention; Figure 3 is a traffic flow change diagram of a certain area in an embodiment of the present invention; Figure 4 are the experimental results of different strategies in the embodiment of the present invention when the traffic volume is low; Figure 5 These are the experimental results of different strategies when the traffic volume is high in the embodiment of the present invention. DETAILED DESCRIPTION

[0017] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0018] like Figure 1As shown in FIG. 1 , the multi-agent visual data unloading method based on information perception and reinforcement learning includes the following steps: Step S1: Collect vehicle status and task data based on edge computing devices, process the data set, and construct training set data.

[0019] Step S11: Each vehicle that enters the communication range of the edge computing device is regarded as an intelligent entity. The vehicle first obtains a visual image and records the time when the image is obtained; based on the edge computing device, the relevant information of the image processing task generated by the vehicle connected to it at the current moment is obtained.

[0020] Among them, the relevant information of the image processing task generated by the vehicle at the current moment includes: the two-dimensional information entropy of the image, the original image file, and the number of vehicles that the edge computing device can connect to at the same time. The two-dimensional information entropy of the image can be used to evaluate the complexity and information content of the image; the original image file can be used for subsequent processing.

[0021] Step S12: After completing information collection, the edge computing device processes each image with different compression ratios. The specific compression ratios are: 1.0, 0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2 and 0.1.

[0022] For each compression ratio, the edge computing device records the information entropy of each image, different compression ratios and corresponding compression time, the processing time of the YOLO model, and the image recall rate after processing by the YOLO model.

[0023] Among them, the information entropy of each image can be used to analyze the impact of different compressions on image information; the processing time of the YOLO model can be used to evaluate the processing speed of the model under different image qualities; the image recall rate after processing by the YOLO model can be used to evaluate the recognition accuracy of the image after being processed by the YOLO model.

[0024] Step S13: The edge computing device generates the state required for Deep Q-Network (DQN) training based on the image information sent by different intelligent agents, and records the accuracy of each image after being processed by a specific application and the total time taken for the image to be generated and processed, so as to facilitate subsequent training of the DQN in the edge computing device.

[0025] Based on the above systematically collected and processed data, comprehensive basic data is provided for subsequent model training to optimize the overall performance of image compression and processing, and finally the training set data is formed, and its data format is shown in Table 1.

[0026] Table 1 Data format of training set ;

[0027] Step S2: Train the multi-agent reinforcement learning network based on DQN to build a DQN model for decision-making.

[0028] Given the dynamic nature of vehicle networks, where multiple vehicles (agents) and an edge server interact to process different volumes of visual data, similar to the complex dynamic problem P that traditional dynamic optimization methods strive to solve, a distributed framework is constructed based on multi-agent reinforcement learning (MARL), such as Figure 2 shown.

[0029] Each vehicle, as a separate agent, learns to make optimal decisions by interacting with the environment and other agents. Therefore, P is represented as a multi-agent stochastic game process, which is solved using MARL. In addition, the design of the multi-agent system also allows new agents to be plugged into the system, making it highly scalable in dynamic VFC environments.

[0030] Among them, the training process of the multi-agent reinforcement learning network based on DQN is as follows: Step S21. First, define the state as a triple (e, s, F); wherein e represents the information entropy of the image, reflecting the complexity and amount of information of the image; s represents the original size of the image, representing the size of the visual data captured by the vehicle set; F represents the computing resource load on the edge computing node, representing the number of vehicles simultaneously connected to the edge computing device at the current moment.

[0031] Step S22: define the action space action, as shown below: ; Among them, 1, 0.9, 0.8, ..., 0.1 represent the image compression ratio.

[0032] Step S23: With the goal of minimizing the time consumption T and maximizing the recognition accuracy G of all images, define the reward of reinforcement learning as follows: ; in, Represents the average recognition accuracy of all images processed by the edge computing node. Represents the average total processing time of all images processed by the edge computing node, that is, the sum of compression time and model processing time; is a weight factor used to adjust the DQN model and degree of emphasis.

[0033] Step S24, therefore, the final optimization goal of DQN reinforcement learning is as follows: ; In the present invention, the state is continuous, and the use of tables to store state values ​​or action values ​​in traditional reinforcement learning is no longer suitable for today's problems. Therefore, in the present invention, based on DQN, the value function is approximated to calculate the action value. The core of DQN is to use the Q function to make decisions. Among them, in order to speed up the training process, the k-step average reward is incorporated into the normal DQN, and its training process is shown in Table 2.

[0034] Table 2. Training process of deep Q-network ; The present invention introduces image information entropy, image original size, and the number of vehicles connected to the edge device at the same time as the decision basis, so that each vehicle can flexibly adjust the image compression level according to the real-time computing resource load. This innovation significantly reduces the transmission cost of redundant visual data and improves the overall efficiency of visual data processing.

[0035] When the vehicle is connected to the edge computing node, it can quickly select the optimal compression ratio and quickly unload important visual information to ensure that key data reaches the processing node in the shortest time. This not only reduces the consumption of vehicle transmission bandwidth, but also improves service response speed while ensuring service quality, solving the contradiction between time consumption and recognition accuracy in traditional methods.

[0036] Step S3: After the DQN training is completed, the edge computing device distributes the trained DQN model to each vehicle that enters its communication range. After the vehicle obtains the trained DQN model, it makes its own decision to determine the compression ratio of the image sent to the edge computing device.

[0037] Step S31: When the vehicle has a new visual data processing task, the computing device on the vehicle calculates the information entropy of the image and the original size of the image. At the same time, the vehicle sends information to the edge computing device to obtain the total number of vehicles connected to it.

[0038] Step S32: Then, the information entropy of the image, the original size of the image, and the total number of vehicles connected to the edge computing device are used as states and input into the trained DQN model. DQN makes corresponding decisions based on the states and gives the optimal compression ratio for the image.

[0039] Step S33: Finally, the vehicle adjusts the resolution of the image according to the optimal compression ratio given by DQN, and then sends the image with adjusted resolution to the edge computing device for subsequent processing.

[0040] In addition, there will be obvious differences in the amount of traffic in different areas at different times. When the traffic volume changes at different times of the day, the DQN networks in the edge computing devices in different areas can transmit to each other.

[0041] For example, area A currently has a DQN network that can cope with high traffic flow, and area B currently has a DQN model that can cope with low traffic flow. When the traffic flow in A and B changes, areas A and B can exchange each other's DQN models, thereby reducing the number of training times, speeding up the transition of compression plans in the two places, and responding to the ever-changing traffic flow changes more quickly and efficiently.

[0042] In addition, edge computing devices can train DQN models under different traffic flow pressures for different traffic flows. When the traffic flow changes, the DQN model assigned to the vehicle can be switched to ensure the efficiency of the compression scheme.

[0043] In this process, the present invention adopts experience-based MARL to give each vehicle the ability to make autonomous decisions, so that it can flexibly respond to complex traffic conditions in a fully observable environment. It does not rely on traditional complex mathematical modeling, but learns through historical experience and automatically adjusts the unloading strategy, so that the system can quickly adapt to the dynamically changing traffic environment. When the traffic is dense or in an emergency, the data unloading strategy is optimized in time, which improves the stability and reliability of the entire intelligent driving system. In addition, the information sharing mechanism between vehicles also enhances the overall collaborative combat capability and further improves traffic safety.

[0044] Example This embodiment is based on the road traffic data of a certain area for experimentation. The traffic flow changes are as follows: Figure 3 As shown, the performance of five strategies including the strategy proposed by the present invention is compared, as shown below: (1) NC: No compression is performed on the visual images captured by each vehicle.

[0045] (2) Random: A random compression ratio is applied to each visual image captured by a vehicle.

[0046] (3) IOVE-T: In multi-agent reinforcement learning training, setting =0.2, which means that the IOVE strategy at this time focuses on the time part , that is, the IOVE strategy tends to reduce the overall service time.

[0047] (4) IOVE-B: In multi-agent reinforcement learning training, setting =0.7, which means that the IOVE strategy at this time focuses on the profit part , that is, the IOVE strategy tends to make the overall recognition accuracy of the image higher.

[0048] (5) IOVE-A: In multi-agent reinforcement learning training, setting =0.9, which means that the IOVE strategy at this time is more focused on the profit part than IOVE-B , that is, the IOVE strategy is more inclined to make the overall recognition accuracy of the image higher than that of IOVE-B.

[0049] like Figure 4 and Figure 5 Figure 2 shows the performance of different strategies in terms of time and accuracy in low-traffic and high-traffic scenarios. In both cases, NC has the highest accuracy but the highest time cost. In contrast, IOVE-T has the lowest average processing time in both cases but also the lowest accuracy.

[0050] Figure 3 and Figure 4 The results show that IOVE-T, IOVE-B, and IOVE-A show consistent trends in both cases. Specifically, as β increases, the accuracy improves while the time cost increases. However, the impact on time is more obvious in the high-traffic case.

[0051] In low-traffic scenarios, the average processing time of IOVE-A is only 0.3 seconds longer than that of IOVE-T, while in high-traffic scenarios the difference increases to 0.64 seconds. This difference is due to the increased number of vehicles and multi-tasking in high-traffic conditions, which results in a more significant time cost.

[0052] In low-traffic scenarios, IOVE-B and IOVEA achieved accuracies of 0.48 and 0.53, respectively, which are higher than Randon’s 0.38. In high-traffic scenarios, the accuracies of IOVE-B and IOVE-A are both better than Random. It is worth noting that in high-traffic scenarios, the accuracy of IOVE-A is close to the highest accuracy of NC, with a difference of only 0.02, but the processing time is saved by an average of 0.13s. This proves the effectiveness of the IOVE algorithm in making intelligent decisions based on image entropy and size, thereby optimizing the compression ratio.

[0053] This embodiment verifies the significant advantages of the IOVE algorithm in visual data offloading efficiency and recognition accuracy through simulation evaluation on real-world traffic data, which not only improves the performance of existing intelligent driving systems, but also opens up new market doors for intelligent driving applications of low-cost vehicles. Traditional high-end models usually require expensive computing hardware, but the present invention reduces the dependence on these resources, which means that more consumers can enjoy the convenience and safety brought by intelligent driving.

[0054] Therefore, the present invention adopts the above-mentioned multi-agent visual data unloading method based on information perception and reinforcement learning, based on DQN reinforcement learning, so that each vehicle can flexibly adjust the compression level of the image according to the real-time computing resource load, and automatically adjust the unloading strategy, so that the system can quickly adapt to the dynamically changing traffic environment, balance the time consumption and processing accuracy of visual data processing in a resource-constrained vehicle environment, improve the performance of existing intelligent driving systems, and open up a new market door for intelligent driving applications of low-cost vehicles.

[0055] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A multi-agent visual data unloading method based on information perception and reinforcement learning, characterized in that: The following steps are involved: Step S1: Collect vehicle status and task data based on edge computing devices, process the data set, and construct training set data; Step S2: training a multi-agent reinforcement learning network based on a deep Q network to construct a DQN model for decision making; Step S3: After the deep Q network training is completed, the edge computing device distributes the trained DQN model to each vehicle that enters its communication range. After the vehicle obtains the trained deep Q network, it makes decisions independently to determine the compression ratio of the image sent to the edge computing device.

2. The multi-agent visual data unloading method based on information perception and reinforcement learning according to claim 1 is characterized in that: In step S1, the specific process of training set data collection and processing is as follows: Step S11: Each vehicle that enters the communication range of the edge computing device is regarded as an intelligent agent. The vehicle first obtains a visual image and records the time when the image is obtained; Based on the edge computing device, obtain relevant information about the image processing task generated by the vehicle connected to it at the current moment; Step S12: After completing the information collection, the edge computing device processes each image with different compression ratios. The specific compression ratios are: 1.0, 0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2 and 0.

1. Step S13: The edge computing device generates the state required for deep Q network training based on the image information sent by different intelligent agents, and records the accuracy of each image after being processed by a specific application and the total time taken for the image to be generated and processed, so as to facilitate subsequent training of the DQN in the edge computing device.

3. The multi-agent visual data unloading method based on information perception and reinforcement learning according to claim 2 is characterized in that: In step S11, the relevant information of the image processing task generated by the vehicle at the current moment includes the two-dimensional information entropy of the image, the original image file, and the number of vehicles that the edge computing device can connect to at the same time; Among them, the two-dimensional information entropy of the image is used to evaluate the complexity and information content of the image; the original image file is used for subsequent processing.

4. The multi-agent visual data unloading method based on information perception and reinforcement learning according to claim 2 is characterized in that: In step S12, for each compression ratio, the edge computing device records the information entropy of each image, different compression ratios and corresponding compression time, the processing time of the YOLO model, and the image recall rate after the YOLO model processes; Among them, the information entropy of each image is used to analyze the impact of different compressions on image information; the processing time of the YOLO model is used to evaluate the processing speed of the model under different image qualities; the image recall rate after processing by the YOLO model is used to evaluate the recognition accuracy of the image after being processed by the YOLO model.

5. The multi-agent visual data unloading method based on information perception and reinforcement learning according to claim 1 is characterized in that: In step S2, the training process of the DQN-based multi-agent reinforcement learning network is as follows: Step S21, first, define the state state as a triple (e, s, F); where e represents the information entropy of the image, reflecting the complexity and information content of the image; s represents the original size of the image, representing the visual data size of the visual data captured by the vehicle set; F represents the computing resource load on the edge computing node, representing the number of vehicles simultaneously connected to the edge computing device at the current moment; Step S22: define the action space action, as shown below: ； Among them, 1, 0.9, 0.8, ..., 0.1 represent the image compression ratio; Step S23: With the goal of minimizing the time consumption T and maximizing the recognition accuracy G of all images, define the reward of reinforcement learning as follows: ； in, Represents the average recognition accuracy of all images processed by the edge computing node. Represents the average total processing time of all images processed by the edge computing node, that is, the sum of compression time and model processing time; is a weight factor used to adjust the DQN model and the degree of emphasis; Step S24, therefore, the final optimization goal of DQN reinforcement learning is as follows: 。 6. The multi-agent visual data unloading method based on information perception and reinforcement learning according to claim 1 is characterized in that: In step S3, the trained DQN model is distributed to each vehicle that enters its communication range, and the specific process of the vehicle making its own autonomous decision is as follows: Step S31: When the vehicle has a new visual data processing task, the computing device on the vehicle calculates the information entropy of the image and the original size of the image, and the vehicle sends information to the edge computing device to learn the total number of connected vehicles; Step S32: Then, the information entropy of the image, the original size of the image, and the total number of vehicles connected to the edge computing device are used as states and input into the trained DQN model. Then, a corresponding decision is made based on the states to give the optimal compression ratio for the image. Step S33: Finally, the vehicle adjusts the resolution of the image according to the optimal compression ratio given by the DQN model, and then sends the image with adjusted resolution to the edge computing device for subsequent processing.

7. The multi-agent visual data unloading method based on information perception and reinforcement learning according to claim 6 is characterized by: Since the volume of traffic in different areas at different times can be significantly different, at different times of the day, when the volume of traffic changes, the deep Q networks in edge computing devices in different areas can achieve mutual transmission.

8. The multi-agent visual data unloading method based on information perception and reinforcement learning according to claim 7 is characterized in that: The edge computing device trains deep Q network models under different traffic flow pressures for different traffic flows. When the traffic flow changes, the deep Q network model assigned to the vehicle is switched to ensure the efficiency of the compression solution.

Images