Action strategy model construction method and device, storage medium and program product

Abstract

The invention relates to the technical field of strategy optimization, and discloses an action strategy model construction method and device, a storage medium and a program product.The action strategy model construction method comprises the steps that object recognition is conducted on target data through a recognition model, recognition features of multiple target objects are obtained, and the target data comprise target image data of a target scene; classifying the target object based on the recognition features to obtain object category features of the target object; performing behavior detection on the recognition features by using an anomaly detection model to obtain behavior features of the target object, the behavior features including conventional behavior features and abnormal behavior features; an object library is constructed based on the object category features and the behavior features, a corresponding strategy library is constructed for the object library, and the strategy library comprises a plurality of preset behavior strategies; and training the initial strategy model based on an object library and a strategy library according to the job tasks to obtain an action strategy model corresponding to each job task. According to the invention, response measures can be quickly provided.

Description

Technical Field

[0001] The present invention relates to the technical field of strategy optimization, and in particular to an action strategy model construction method, device, storage medium and program product. Background Art

[0002] The changeable weather in the wild, such as high temperature, severe cold, heavy rain, strong wind, etc., will increase the difficulty of operation. The complex terrain in the wild, such as mountains and forests, will increase the difficulty of travel and operation. For example, when conducting geological surveys in mountainous areas, you need to face steep slopes and rugged roads. Wild animals in certain areas of the wild may pose a threat to workers.

[0003] Construction work and maintenance are mostly carried out in the field. Because the field working environment is complex and changeable, workers often face various risks such as wildlife intrusion, natural disasters such as landslides and floods, and failure of operating equipment.

[0004] The current traditional manual monitoring method is difficult to fully cover all potential dangers in real time. When environmental changes, dangerous objects or dangerous events are detected, it is impossible to respond quickly and effectively, and it is difficult to ensure the safety of operators. There are problems such as insufficient real-time monitoring capabilities, insufficient early warning, single response measures and poor adaptability. Summary of the invention

[0005] In view of this, the present invention provides a method, device, storage medium and program product for constructing an action strategy model to solve the problem of being unable to provide response measures quickly.

[0006] In a first aspect, the present invention provides a method for constructing an action strategy model, which includes: using a recognition model to perform object recognition on target data to obtain recognition features of multiple target objects, the target data includes target image data of a target scene; classifying the target objects based on the recognition features to obtain object category features of the target objects; using an anomaly detection model to perform behavior detection on the recognition features to obtain behavior features of the target objects, the behavior features include regular behavior features and abnormal behavior features; constructing an object library based on the object category features and the behavior features, and constructing a corresponding strategy library for the object library, the strategy library including multiple preset behavior strategies; according to the work task, training the initial strategy model based on the object library and the strategy library to obtain an action strategy model corresponding to each work task.

[0007] In this implementation, multiple object types and object abnormal features are determined based on target recognition, target classification and anomaly detection, and an object library is built based on object feature information. The object library can measure object features from two dimensions: category and whether it is abnormal; on this basis, multiple preset behavior strategies are set, and according to business tasks, the corresponding object features and preset behavior strategies are matched using the strategy model to respond to different changes in object behavior. The action strategy model provided in this application can automatically output the most suitable action strategy based on the current object action features, and can quickly provide response measures.

[0008] In an optional embodiment, the target data also includes environmental data of the target scene. According to the work task, the initial strategy model is trained based on the object library and the strategy library to obtain the action strategy model corresponding to each work task, and then the following steps are further included: fusing the environmental data and the identification features to obtain the environmental features of multiple target objects; using the object library and the environmental features as features and the strategy library as labels to perform secondary training on the action strategy model to obtain an environmental action strategy model.

[0009] In this implementation, the impact of the environment on the strategy is considered, the environmental data and identification features are integrated to obtain environmental features that characterize the impact of the environment, and the environmental features and object features are combined as input features to perform secondary training on the action strategy model. The action strategy model after the first training can quickly respond to the impact of the object's own characteristics, and the environmental action strategy model after the second training can further respond to the combined impact of the environment and the object's own characteristics, providing a response strategy that is more suitable for the actual situation and improving safety assurance.

[0010] In an optional implementation, the action strategy model is an Actor-Critic model, which includes an Actor network and a Critic network. The object library and environmental features are used as features, and the strategy library is used as a label. The action strategy model is trained twice to obtain an environmental action strategy model, including: using the object library and environmental features as input features of the Actor network, and calculating the probability distribution of each preset behavior strategy; using the object library and environmental features as input features of the Critic network, and calculating the value function of each preset behavior strategy; determining the target strategy based on the probability distribution and the value function; using the strategy library as the label of the target strategy, and adjusting the parameters of the Actor network and the Critic network based on the loss function to obtain the environmental action strategy model.

[0011] In an optional implementation, adjusting the parameters of the Actor network and the Critic network based on the loss function to obtain the environment action strategy model includes: adjusting the parameters of the Actor network using the loss function of proximal strategy optimization; adjusting the parameters of the Critic network using the loss function of temporal difference to obtain the environment action strategy model.

[0012] In this implementation, an Actor-Critic model is constructed to adapt to different work tasks and environmental conditions. It can provide strategies efficiently and flexibly. Combined with strategy iteration and environmental adaptability enhancement in reinforcement learning, it can adjust strategy parameters according to environmental changes and actual work feedback, thereby achieving continuous optimization and personalized configuration of strategies.

[0013] In an optional embodiment, before using the recognition model to perform object recognition on the target data, it also includes: performing an image enhancement process on the initial image data based on the PDCE-Net network to obtain first enhanced image data; wherein the PDCE-Net network includes a PCov convolution module, and the activation function of the PDCE-Net network is a LeakyReLU function; performing gamma correction on the initial image data to obtain corrected image data; performing a secondary image enhancement process on the initial image data based on the PRED-Net network, and during the processing, fusing the first enhanced image data and the corrected image data to obtain the target image data.

[0014] In this implementation, by enhancing the image data, problems such as low contrast, low signal-to-noise ratio and underexposure can be solved, the computing overhead of the equipment can be reduced, the image processing efficiency can be improved, the real-time requirements of dynamic targets can be met, the image brightness can be improved, the image color can be balanced, and a basis for subsequent information collection can be provided.

[0015] In an optional embodiment, using a recognition model to perform object recognition on target data to obtain recognition features of multiple target objects includes: modifying the PANFPN structure of the YOLOv8 model to a RepGFPN structure, and adding an attention mechanism module before the SPPF module to obtain a SA-YOLOv8 model; performing object recognition on the target data based on the SA-YOLOv8 model to obtain recognition features.

[0016] In this implementation, the YOLOv8 model is modified so that the recognition model can improve its feature processing capabilities and has a stronger ability to transmit semantic information across levels.

[0017] In an optional embodiment, the anomaly detection model is used to perform behavior detection on the identification features to obtain the behavior characteristics of the target object, including: using parallel computing technology to construct multiple isolation forests, and using the isolation forests to detect the identification features separately; calculating the importance of the identification features based on the path lengths of the multiple isolation forests, and obtaining normal behavior characteristics and abnormal behavior characteristics based on the importance.

[0018] In this implementation, unsupervised learning is performed on the data to identify potential anomalies and evaluate the importance of features. The subsequent strategy selection can focus on key features to ensure the targetedness and effectiveness of the selected strategy.

[0019] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the action strategy model construction method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0020] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the action strategy model construction method of the first aspect or any corresponding embodiment thereof.

[0021] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the action strategy model construction method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 is a flow chart of a method for building an action strategy model according to an embodiment of the present invention;

[0024] Figure 2 is a flow chart of another method for building an action strategy model according to an embodiment of the present invention;

[0025] Figure 3 is a flowchart of another method for building an action strategy model according to an embodiment of the present invention;

[0026] Figure 4is a schematic diagram of the structure of a target image data network according to an embodiment of the present invention;

[0027] Figure 5 is a structural block diagram of an action strategy model building device according to an embodiment of the present invention;

[0028] Figure 6 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0030] In order to address the challenges of personal safety and work efficiency in complex and changeable outdoor environments, this application proposes a method for constructing an action strategy model. Through a machine learning model, it detects possible dangerous events, such as approaching wild animals, signs of landslides, flood warnings, etc., and provides corresponding action strategies in a timely manner.

[0031] According to an embodiment of the present invention, an embodiment of an action strategy model construction method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0032] In this embodiment, a method for constructing an action strategy model is provided. Figure 1 is a flow chart of a method for constructing an action strategy model according to an embodiment of the present invention. It should be noted that if there are substantially the same results, this embodiment does not use Figure 1 The process sequence shown is limited. Figure 1 As shown, the process includes the following steps:

[0033] Step S101, using a recognition model to perform object recognition on target data to obtain recognition features of multiple target objects.

[0034] The target data includes target image data of the target scene.

[0035] The target image data is collected by using an image collection device, and the recognition model is used to perform object recognition on the target image data to detect and obtain recognition features of multiple objects.

[0036] Among them, the recognition models include R-CNN (Region-based Convolutional Neural Network) model, YOLO (You Only Look Once) model, SSD (Single Shot MultiBox Detector) model, etc.

[0037] In an optional implementation, the YOLOv8 model is used to perform object recognition on the target image data, a detection frame containing the target object is obtained, and features of the object in the detection frame are extracted to obtain recognition features of the target object.

[0038] Step S102: classify the target object based on the recognition feature to obtain the object category feature of the target object.

[0039] The target object is classified using the identification features obtained by detection, and the object category features of the target object are detected.

[0040] Among them, the object category features include plant targets, animal targets, terrain targets, artificial facility targets, etc. More specifically, plant targets, animal targets, terrain targets, and artificial facility targets are classified in detail. For example, artificial facility targets include roads, buildings, equipment, etc.

[0041] Exemplarily, characteristic attributes corresponding to each object category are determined, such as shape, color, texture, etc. Objects are classified according to the characteristics.

[0042] Step S103: Use the anomaly detection model to perform behavior detection on the recognition features to obtain the behavior features of the target object.

[0043] Among them, behavioral characteristics include regular behavioral characteristics and abnormal behavioral characteristics.

[0044] Among them, anomaly detection models include cluster analysis algorithms, Isolation Forest models, SVM (Support Vector Machine) models, etc.

[0045] In an optional implementation, the isolation forest model is used to perform further abnormal data detection on the object category features detected by the recognition model to detect whether the behavior of the target object is abnormal and obtain the behavior characteristics of the target object.

[0046] On the basis of classifying the target objects by using the object category features, the target objects are further classified according to whether the behavior of the target objects is abnormal.

[0047] Step S104: construct an object library based on the object category features and behavior features, and construct a corresponding strategy library for the object library.

[0048] The object library includes multiple sets of object category features and behavior features. Specifically, each object category feature is combined with its corresponding multiple behavior features to obtain multiple sets of object category features and behavior features. The object library is constructed based on the object category features and behavior features of multiple objects.

[0049] For example, for object a, the method of step S101 to step S103 is used to identify object a multiple times and obtain that object a has behavior feature 1, behavior feature 2 and behavior feature 3. The object and the behavior feature are combined to obtain (object a, behavior feature 1), (object a, behavior feature 2), (object a, behavior feature 3). An object library is constructed based on multiple behavior features of multiple objects.

[0050] The strategy library includes multiple preset behavior strategies. Specifically, a series of preset behavior strategies are designed for each object category. The preset behavior strategies cover the entire process from object recognition to action, such as adopting avoidance strategies when encountering wild animals, and collecting samples when identifying specific plants.

[0051] In one possible implementation, the preset behaviors and strategies include early warnings, such as reminding workers to pay attention to safety during primary warnings, initiating local evacuation plans during intermediate warnings, and immediately executing full emergency evacuations during high-level warnings, and automatically cutting off the power supply to high-risk operating equipment to ensure the safety of personnel lives.

[0052] Step S105 , according to the operation task, the initial strategy model is trained based on the object library and the strategy library to obtain the action strategy model corresponding to each operation task.

[0053] According to the operation task, the object library is used as the input feature and the strategy library is used as the label to train the initial strategy model to obtain the action strategy model corresponding to each operation task. According to the preset risk level and the actual situation on site, the corresponding warning level and response measures are automatically triggered.

[0054] The action strategy model construction method provided in this embodiment determines multiple object types and object abnormal features based on target recognition, target classification and anomaly detection, and constructs an object library based on object feature information. The object library can measure object features from two dimensions: category and whether it is abnormal; on this basis, multiple preset behavior strategies are set, and according to business tasks, the corresponding object features and preset behavior strategies are matched using the strategy model to respond to different changes in object behavior. The action strategy model provided in this application can automatically output the most suitable action strategy based on the current object action features, and can provide response measures quickly.

[0055] In this embodiment, a method for constructing an action strategy model is provided. Figure 2 is a flowchart of another method for constructing an action strategy model according to an embodiment of the present invention. It should be noted that if there are substantially the same results, this embodiment does not use Figure 2 The process sequence shown is limited. Figure 2 As shown, the process includes the following steps:

[0056] Step S201, using a recognition model to perform object recognition on target data to obtain recognition features of multiple target objects.

[0057] The target data includes target image data of the target scene.

[0058] Specifically, the above step S201 includes:

[0059] Step S2011, modify the PANFPN structure of the YOLOv8 model to a RepGFPN structure, and add an attention mechanism module before the SPPF module to obtain the SA-YOLOv8 model.

[0060] This application pre-builds the YOLOv8 model and further modifies it based on the YOLOv8 model.

[0061] The YOLOv8 model includes a backbone network, a neck network, and a head network. The backbone network and the neck network of the YOLOv8 model are modified to obtain the SA-YOLOv8 model.

[0062] Specifically, adding a Shuffle Attention (SA) module before the SPPF module of the backbone network can improve the feature recognition capability of the target object. In the neck network, the PANFPN structure is modified to the RepGFPN structure, which can enhance the model's ability to transfer semantic information across levels.

[0063] Step S2012: perform object recognition on the target data based on the trained SA-YOLOv8 model to obtain recognition features.

[0064] Fine-tune the SA-YOLOv8 model using the loss function.

[0065] Specifically, the SA-YOLOv8 model is fine-tuned using the cross entropy loss function and the positioning error loss function. The loss function is:

[0066]

[0067] Among them, L cls is the classification loss, L box is the bounding box regression loss, L obj is the loss of object existence, L iouis the IoU loss. obj Reflects the confidence loss of whether the predicted box contains the object, L iou The loss is calculated based on the IoU value between the predicted box and the true box. It is only used when the predicted box is marked as containing an object. This helps improve the accuracy of bounding box positioning; L obj and L iou is the corresponding loss weight, which is used to balance the importance of each loss.

[0068] After training is completed, the SA-YOLOv8 model is used to perform object recognition on the target data to obtain recognition features.

[0069] In order to solve the problems of difficulty in extracting low-light image features, low detection accuracy and slow recognition speed, this application uses the SA-YOLOv8 model to identify objects in target data, which can improve the reliability of target detection in low-light environments, achieve accurate object recognition, and ensure that the model can accurately identify specific objects by fine-tuning and optimizing the loss function. It can provide key information for subsequent situation understanding and strategy formulation.

[0070] Step S202: classify the target object based on the recognition feature to obtain the object category feature of the target object.

[0071] For details, please see Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.

[0072] Step S203: Use the anomaly detection model to perform behavior detection on the recognition features to obtain the behavior features of the target object.

[0073] Among them, behavioral characteristics include regular behavioral characteristics and abnormal behavioral characteristics.

[0074] Specifically, the above step S203 includes:

[0075] Step S2031, using parallel computing technology to construct multiple isolation forests, and using the isolation forests to detect the identification features respectively.

[0076] Among them, parallel computing technology includes Python joblib technology.

[0077] Specifically, parallel computing technology is used to simultaneously create and run multiple isolation forest classification trees.

[0078] On large data sets, by having P processors working in parallel, P trees can be built in parallel in each round, reducing the total computing time and maintaining algorithm efficiency and optimal utilization of resources.

[0079] Step S2032, calculating the importance of the identification features based on the path lengths of the multiple isolation forests, and obtaining the normal behavior features and the abnormal behavior features based on the importance.

[0080] The importance of identification features is evaluated using an extended method, and the contribution of each identification feature to anomaly detection is quantified.

[0081] Specifically, the average path length change caused by considering a single feature in each isolation forest is calculated and normalized to obtain the importance of the identification feature.

[0082] Among them, the average path length changes as follows:

[0083]

[0084] Where, ΔE t,j is the average path length of t trees when only considering the jth identification feature, E t is the average path length when the t-th tree does not specifically distinguish features.

[0085] Among them, the importance is:

[0086]

[0087] Among them, M is the total number of features, ensuring that the sum of the importance of all identification features is 1.

[0088] Set anomaly thresholds and perform unsupervised learning on environmental data.

[0089] Furthermore, the subsampling ratio and the maximum depth of the tree are set. Specifically, the sampling ratio is set according to the size and complexity of the data.

[0090] Among them, large data sets focus on acceleration, while small data sets prioritize accuracy. At the same time, the growth of the tree is limited by the early stopping strategy. When the number of node samples is lower than the preset threshold, the splitting is stopped, effectively controlling the complexity of the model and improving the operation efficiency.

[0091] In this implementation, unsupervised learning of data can identify potential anomalies and evaluate feature importance, so that the subsequent strategy selection can focus on key features to ensure the pertinence and effectiveness of the strategy selection. Anomaly detection can enhance the ability to identify unknown or rare objects.

[0092] Step S204: construct an object library based on the object category features and behavior features, and construct a corresponding strategy library for the object library.

[0093] For details, please see Figure 1 Step S104 of the illustrated embodiment will not be described in detail here.

[0094] Step S205 , according to the operation task, the initial strategy model is trained based on the object library and the strategy library to obtain the action strategy model corresponding to each operation task.

[0095] The initial policy model is the Actor-Critic model, which includes the Actor network and the Critic network. The Actor network is used to learn what preset behavior strategy to adopt in a given state, and the Critic network is used to evaluate the preset behavior strategy. The initial policy model is trained with the goal of optimizing decisions in the continuous action space.

[0096] Among them, the Actor-Critic model can combine the stability and sample efficiency of the value function method with the flexibility of the policy gradient method. The Actor-Critic model can make decisions in the continuous action space of complex tasks, such as automated management of field operations.

[0097] Specifically, the above step S205 includes:

[0098] Step S2051, using the object library as the input feature of the Actor network, and calculating the probability distribution of each preset behavior strategy.

[0099] Build an Actor network and input data into the database as the current state s t , output the probability distribution π(a t |s t θ π ), where θ π Parameters for the Actor network.

[0100] Step S2052, using the object library as the input feature of the Critic network, and calculating the value function of each preset behavior strategy.

[0101] Construct a Critic network, and input the database as the current state s t , output the state value function V(s t θ V ) or the action value function Q(s) of the preset behavior strategy t ,a t θ Q ), where θ V and θ Q are the parameters of the Critic network.

[0102] Step S2053, determine the target strategy based on the probability distribution and the value function, use the strategy library as the label of the target strategy, adjust the parameters of the Actor network and the Critic network based on the loss function, and obtain the action strategy model.

[0103] Based on the loss function, the initial Actor-Critic model is updated and the model parameters are adjusted to obtain the action strategy model.

[0104] Furthermore, the present application considers the impact of environmental factors on strategy selection and provides a method for constructing an action strategy model in this embodiment. Figure 3 is a flowchart of another method for constructing an action strategy model according to an embodiment of the present invention. It should be noted that if there are substantially the same results, this embodiment does not use Figure 3 The process sequence shown is limited. Figure 3 As shown, the process includes the following steps:

[0105] Step S301, obtaining target data.

[0106] The target data includes target image data and environmental data of the target scene. The environmental data includes temperature data, sound data, humidity data, wind speed data, light intensity data, air pressure data, etc.

[0107] Target data is collected using multi-source sensors, including visual sensors, infrared thermal imagers, sound sensors, and environmental parameter detectors.

[0108] Specifically, visual sensors, including high-definition cameras, with a resolution greater than 1080p are deployed in locations with a wide field of view or in key monitoring locations, and the high-definition cameras are used to collect target image data in real time.

[0109] Place infrared thermal imagers at key monitoring locations, set their waveband to 7-14μm, and use them to detect temperature at night or on hidden target objects to collect temperature data.

[0110] Sound sensors are deployed in areas that are sensitive to noise or require sound warnings. The frequency response range is set to 20Hz-20kHz, covering the audible range of the human ear, and sound data is collected using sound sensors.

[0111] The location sensor collects the target location data provided by GPS data or other positioning systems.

[0112] Motion sensors, including speedometers, gyroscopes and other inertial measurement units, collect motion status data.

[0113] Environmental parameter detectors include temperature and humidity sensors, wind speed sensors, light intensity sensors, and air pressure sensors, which monitor environmental data in all directions to obtain temperature and humidity data, wind speed data, light intensity data, air pressure data, etc.

[0114] Step S302, preprocessing the target data.

[0115] Among them, the preprocessing includes denoising, standardization and enhancement of the target image data.

[0116] First, the target image data is denoised.

[0117] The high-frequency noise in the target image data is removed by low-pass filtering, where the noise filtering is:

[0118] y(t)=x(t)-α(x(t)-x(t-1)).

[0119] Among them, x(t) is the target data before filtering, y(t) is the target data after filtering, and α is the attenuation coefficient.

[0120] Furthermore, the target image data after denoising is normalized or standardized, wherein the normalization calculation formula is:

[0121]

[0122] The calculation formula for standardization is:

[0123]

[0124] Among them, x is the target data before normalization, X min is the minimum value of the target data, X max is the maximum value of the target data, μ is the mean, and σ is the standard deviation.

[0125] This application can ensure the quality of target image data through noise filtering and data standardization, provide clean and standardized input for subsequent models, and enhance the efficiency and accuracy of model learning.

[0126] Furthermore, the standardized target image data is enhanced based on the PDCE-Net network and the PRED-Net network.

[0127] Specifically, see Figure 4 , Figure 4 FIG. 1 is a schematic diagram of a target image data network structure according to an embodiment of the present invention. The network structures of the PDCE-Net network and the PRED-Net network are as follows: Figure 4 shown.

[0128] Firstly, the initial image data is enhanced based on the PDCE-Net network to obtain the first enhanced image data.

[0129] Specifically, the DCE network is improved by replacing the original depth-separable convolution with some PCov convolutions and replacing the ReLU activation function with the LeakyReLU activation function to obtain the PDCE-Net network, which outputs the first enhanced image data through the light intensity curve iteration. Among them, the PDCE-Net network has the advantage of unsupervised training of the DCE network, so that it can complete network training without a light and dark paired data set. At the same time, the lighter convolution can reduce the computational overhead of the computing device and improve the image processing efficiency. The LeakyReLU activation function can improve the convergence and generalization ability.

[0130] Gamma correction is performed on the initial image data to obtain corrected image data. The initial image data is subjected to secondary image enhancement processing based on the PRED-Net network. During the processing, the first enhanced image data and the corrected image data are fused to obtain the target image data. While improving the brightness of the target image data, it can further suppress noise and balance colors.

[0131] Due to the low contrast, low signal-to-noise ratio and insufficient exposure of monitoring images in dark environments at night, the monitoring efficiency of night observation and tracking of human behavior trajectories is reduced. This application adopts a heavy enhancement strategy for target image data collected in low-light environments to increase image brightness, balance image colors, improve image processing efficiency, and meet the high real-time requirements in dynamic target monitoring.

[0132] Step S303: perform object recognition on the target image using the recognition model to obtain recognition features of multiple target objects, and classify the target objects based on the recognition features to obtain object category features of the target objects.

[0133] For details, please see Figure 2 Step S202 and step S203 of the illustrated embodiment will not be described in detail here.

[0134] Step S304: Use the anomaly detection model to perform behavior detection on the recognition features to obtain the behavior features of the target object.

[0135] Behavioral characteristics include normal behavioral characteristics and abnormal behavioral characteristics. Figure 2 Step S204 of the illustrated embodiment will not be described in detail here.

[0136] Step S305: fusing the environmental data and the identification features to obtain environmental features of multiple target objects.

[0137] The Kalman filtering method is used to fuse environmental data and identification features to obtain the environmental characteristics of multiple target objects.

[0138] Among them, the recognition features include information such as object type, position, size, etc. recognized by the visual sensor.

[0139] The environmental data includes other data detected by various other sensors, such as temperature sensors, sound sensors, motion sensors, etc.

[0140] Specifically, the Kalman filter method is used to fuse the corresponding identification features of each target object in the identification frame with the environmental data to obtain the environmental features corresponding to each target object. The Kalman filter fusion formula is:

[0141]

[0142] in, is the updated state estimate, K k is the Kalman gain, Z k is the measurement value, and H is the observation matrix.

[0143] Kalman filtering technology is used to fuse multi-source data to generate more accurate environmental state estimates and provide a comprehensive perspective for risk assessment.

[0144] Furthermore, based on the fused environmental features, a Bayesian network model is constructed to evaluate potential risks in operations, provide risk predictions for strategy formulation, and ensure that the action strategy is both efficient and safe.

[0145] Specifically, a Bayesian network model is constructed, node relationships are defined, prior probabilities and conditional probabilities are determined, and risk assessment results are output. The conditional probabilities of nodes in the Bayesian network are used to calculate the probability of an event under given conditions.

[0146] Step S306: According to the operation task, the initial strategy model is trained based on the object library and the strategy library to obtain the action strategy model corresponding to each operation task.

[0147] For details, please see Figure 2 Step S205 of the illustrated embodiment will not be described in detail here.

[0148] Step S307 , taking the object library and the environment features as features and the strategy library as labels, the action strategy model is trained twice to obtain an environment action strategy model.

[0149] Based on the above steps, we get the action strategy model π for different tasks i i , the corresponding Actor network parameters are The action strategy model can reflect the initial preferences and action strategy guidance under different tasks.

[0150] Furthermore, the environmental feature c is added to the input feature t, the environmental features and object library state s i Combined to enhance the current state Considering the impact of the environment on the action strategy, dynamically adjust the input of the Actor network, enhance the current state as the input feature of the Actor network, and output the probability distribution of taking each preset behavior strategy in the current state Likewise, the current state will be enhanced As the input feature of the Critic network, output the state value function in the current state Or the action value function of the preset behavior strategy

[0151] Utilize reinforcement learning algorithms to iteratively optimize strategies, and adjust model parameters based on environmental changes and actual job feedback to achieve continuous optimization and personalized configuration of strategies.

[0152] Specifically, the loss function of Proximal Policy Optimization (PPO) is used to adjust the parameters of the Actor network. The loss function of Proximal Policy Optimization is:

[0153]

[0154] in, is the strategy ratio, is the advantage estimate, ∈ is the shear range.

[0155] Specifically, the loss function of temporal difference (TD) is used to adjust the parameters of the critic network. The loss function of temporal difference is:

[0156]

[0157] Based on the above loss function, the object library and environmental features are used as features, and the strategy library is used as labels. The action strategy model is trained twice to obtain the environmental action strategy model.

[0158] In this implementation, an Actor-Critic model is built to adapt to different tasks and environmental conditions. It can provide strategies efficiently and flexibly, and combined with the strategy iteration and environmental adaptability enhancement in reinforcement learning, it can adjust the strategy parameters according to environmental changes and actual operation feedback, and achieve continuous optimization and personalized configuration of the strategy. And it can dynamically adjust the action strategy in combination with environmental changes and object state changes. For example, avoid approaching certain terrain features prone to landslides on rainy days.

[0159] The environmental action strategy model is applied to field work activities. For the current task i and the current state including environmental data, an appropriate environmental action strategy model is selected to perform the corresponding work task.

[0160] Specifically, a lightweight strategy selection network is designed to score the output strategies of multiple environmental action strategy models, and the environmental action strategy model with the highest score is selected based on the environmental adaptability score.

[0161] In an optional implementation, after each policy task is executed, target object data, environment data, and corresponding policy data are collected, and the execution results are labeled, wherein the labels include success, failure, safety indicators, and the like.

[0162] The model is regularly trained and updated based on the collected historical data, and the model parameters of the environment action strategy model are optimized using online learning or periodic offline training to adapt to current task requirements and environmental changes.

[0163] In an optional implementation, a reward function is constructed to ensure the sustainability and reliability of the task by balancing safety and efficiency in the reward function.

[0164] Specifically, the reward function is:

[0165] R t =r task (s t ,a t )+λ sofe r sofe (s t ,a t )-λ cost r cost (s t ,a t ).

[0166] Among them, r task Reward for completing the task, sofe For safety rewards, cost is the execution cost, λ sofe and λ cost is the weight factor.

[0167] Combined with the Actor-Critic model, personalized configuration and dynamic adjustment of strategies are achieved, allowing the automation system to adapt to various complex situations in field operations more flexibly and efficiently.

[0168] The action strategy model construction method provided in this embodiment considers the impact of the environment on the strategy, integrates the environmental data and the identification features, obtains the environmental features that characterize the environmental impact, combines the environmental features and the object features as input features, and performs secondary training on the action strategy model. The action strategy model after the first training can quickly respond to the impact of the object's own characteristics, and the environmental action strategy model after the secondary training can further respond to the combined impact of the environment and the object's own characteristics, providing a response strategy that is more suitable for the actual situation and improving safety assurance.

[0169] This embodiment also provides a portable smart wearable device for continuously monitoring the physiological indicators and location information of the workers. Once the workers are found to be abnormal or in a potentially dangerous area, personalized safety reminders and rescue signals can be sent immediately. Combined with the drone inspection module, regular or on-demand inspections can be carried out in areas that are difficult for human resources to reach, further expanding the monitoring range and improving the comprehensiveness and accuracy of the early warning system.

[0170] This application uses intelligent technology throughout the process from data collection to strategy execution, significantly improving the efficiency and accuracy of field operations, reducing human errors, and enhancing the scientificity and timeliness of decision-making. A strategy model that includes preset strategies and dynamic adjustment mechanisms has been built, which not only provides a series of basic strategy templates, but also realizes personalized configuration and instant optimization of strategies to adapt to the diverse needs of operational tasks.

[0171] In the strategy design, a comprehensive evaluation system that takes into account both safety and efficiency is introduced. Through a carefully designed reward function, the optimal balance between the two is achieved in the reinforcement learning process. A comprehensive risk assessment model is introduced, combined with an intelligent monitoring and early warning system, to effectively prevent potential accidents and improve the safety factor of operations. The reinforcement learning algorithm is used to continuously iterate the strategy, so that the system has the ability to self-learn and optimize. Over time, the system performance will continue to improve and adapt to a wider range of application scenarios.

[0172] In the present embodiment, an action strategy construction device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.

[0173] This embodiment provides an action strategy construction device, such as Figure 5 As shown, including:

[0174] The recognition module 501 is used to perform object recognition on target data using a recognition model to obtain recognition features of multiple target objects. The target data includes target image data of a target scene.

[0175] The classification module 502 is used to classify the target object based on the recognition feature to obtain the object category feature of the target object.

[0176] The anomaly detection module 503 is used to perform behavior detection on the recognition features using the anomaly detection model to obtain the behavior features of the target object, where the behavior features include regular behavior features and abnormal behavior features.

[0177] The strategy library construction module 504 is used to construct an object library based on the object category characteristics and behavior characteristics, and to construct a corresponding strategy library for the object library, wherein the strategy library includes a plurality of preset behavior strategies.

[0178] The strategy building module 505 is used to train the initial strategy model according to the operation task based on the object library and the strategy library to obtain the action strategy model corresponding to each operation task.

[0179] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0180] The action strategy construction device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0181] The embodiment of the present invention also provides a computer device having the above Figure 5 The action strategy construction device shown.

[0182] See also Figure 6 , Figure 6 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 6 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 6 A processor 10 is taken as an example.

[0183] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0184] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0185] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0186] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0187] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0188] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0189] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0190] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0191] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for constructing an action strategy model, characterized in that: The method comprises: Using the recognition model to perform object recognition on target data to obtain recognition features of multiple target objects, the target data including target image data of a target scene; Classifying the target object based on the identification feature to obtain an object category feature of the target object; Performing behavior detection on the identification features using an anomaly detection model to obtain behavior features of the target object, wherein the behavior features include regular behavior features and abnormal behavior features; Building an object library based on the object category characteristics and the behavior characteristics, and building a corresponding strategy library for the object library, wherein the strategy library includes a plurality of preset behavior strategies; According to the operation task, the initial strategy model is trained based on the object library and the strategy library to obtain an action strategy model corresponding to each operation task.

2. The action strategy model construction method according to claim 1, characterized in that: The target data also includes environmental data of the target scene. The training of the initial strategy model based on the object library and the strategy library according to the operation task to obtain the action strategy model corresponding to each operation task also includes: Fusion of the environmental data and the identification features to obtain environmental features of a plurality of target objects; The object library and the environmental features are used as features, the strategy library is used as a label, and the action strategy model is trained twice to obtain an environmental action strategy model.

3. The method for constructing an action strategy model according to claim 2, characterized in that: The action strategy model is an Actor-Critic model, which includes an Actor network and a Critic network. The object library and the environmental features are used as features, and the strategy library is used as a label. The action strategy model is trained twice to obtain an environmental action strategy model, including: Using the object library and the environmental features as input features of the Actor network, and calculating the probability distribution of each of the preset behavior strategies; Using the object library and the environmental features as input features of the Critic network, and calculating the value function of each of the preset behavior strategies; determining a target strategy based on the probability distribution and the value function; The strategy library is used as the label of the target strategy, and the parameters of the Actor network and the Critic network are adjusted based on the loss function to obtain the environmental action strategy model.

4. The method for constructing an action strategy model according to claim 3, characterized in that: The adjusting the parameters of the Actor network and the Critic network based on the loss function to obtain the environmental action strategy model includes: Adjusting the parameters of the Actor network using a loss function optimized by a proximal strategy; The parameters of the Critic network are adjusted using the loss function of temporal difference to obtain the environmental action strategy model.

5. The method for constructing an action strategy model according to claim 1, characterized in that: Before the object recognition of the target data by using the recognition model, the following steps are also included: Performing an image enhancement process on the initial image data based on the PDCE-Net network to obtain first enhanced image data; wherein the PDCE-Net network includes a PCov convolution module, and the activation function of the PDCE-Net network is a LeakyReLU function; Performing gamma correction on the initial image data to obtain corrected image data; The initial image data is subjected to secondary image enhancement processing based on the PRED-Net network. During the processing, the first enhanced image data and the corrected image data are fused to obtain the target image data.

6. The method for constructing an action strategy model according to claim 1, characterized in that: The object recognition is performed on the target data using the recognition model to obtain recognition features of multiple target objects, including: The PANFPN structure of the YOLOv8 model is modified to the RepGFPN structure, and an attention mechanism module is added before the SPPF module to obtain the SA-YOLOv8 model; Object recognition is performed on the target data based on the SA-YOLOv8 model to obtain the recognition features.

7. The method for constructing an action strategy model according to claim 1, characterized in that: The using of the anomaly detection model to perform behavior detection on the recognition feature to obtain the behavior feature of the target object includes: Constructing multiple isolation forests using parallel computing technology, and using the isolation forests to detect the identification features respectively; The importance of the identification feature is calculated based on the path lengths of the plurality of isolation forests, and the normal behavior feature and the abnormal behavior feature are obtained based on the importance.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the action strategy model construction method according to any one of claims 1 to 7 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the action strategy model construction method according to any one of claims 1 to 7.

10. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the action strategy model building method according to any one of claims 1 to 7.

Images