A task processing method and system based on multi-agent collaboration

Abstract

The present application relates to the technical field of task processing, in particular to a task processing method and system based on multi-agent cooperation, comprising: performing a structured semantic analysis operation on the full volume of text of a target case to obtain a case element semantic vector; performing a multi-scale image analysis operation on the scene panoramic image data of the target case to obtain a basic image feature tensor and a local detail feature tensor, and performing a spatiotemporal correlation feature extraction operation on the local detail feature tensor to obtain a spatiotemporal correlation feature result. Through comprehensive processing of the case volume text, image data and spatiotemporal features, the present application can effectively integrate multi-modal data from different sources, so that police personnel can obtain more comprehensive and accurate information when processing a case. This data fusion enhances the depth and breadth of case analysis and reduces the understanding bias caused by information silos.

Description

Technical Field

[0001] This invention relates to the field of task processing technology, specifically to a task processing method and system based on multi-agent collaboration. Background Technology

[0002] With the acceleration of global urbanization, cities are becoming larger and cities are becoming more densely populated. From public security management to public safety and emergency response to sudden incidents, multiple fields and forces need to work together. This highly complex social environment requires a method for handling police incidents that can be effectively organized and coordinated within a distributed and collaborative framework.

[0003] Currently, traditional methods typically rely on a single data source, lacking effective integration of multimodal data. Police officers often have to make judgments based on limited, single data points. This information silo can lead to incomplete and inaccurate case analysis, causing misunderstandings. Furthermore, it usually relies on manual analysis of case files, manually extracting information from documents, which is not only inefficient but also prone to missing crucial information.

[0004] Furthermore, traditional methods often fail to fully extract potential key information from images when processing on-site image data, especially in multi-scale analysis and spatiotemporal feature extraction. They also rely heavily on manual intervention, such as manual analysis of case files, on-site images, and other data. The analysis process is slow and susceptible to human error. Moreover, they often lack visualization analysis of the spatial distribution and activity trajectories of case elements, making it difficult for police officers to accurately grasp the core elements of the case. Summary of the Invention

[0005] To achieve the above objectives, the present invention provides the following technical solution: a task processing method based on multi-agent cooperation, comprising:

[0006] A structured semantic parsing operation is performed on the full case file text of the target case to obtain the semantic vector of case elements;

[0007] Multi-scale image analysis is performed on the panoramic image data of the target case to obtain the basic image feature tensor and the local detail feature tensor. Spatiotemporal correlation feature extraction is performed on the local detail feature tensor to obtain the spatiotemporal correlation feature results.

[0008] The semantic vectors of case elements, the basic image feature tensors, and the spatiotemporal correlation feature results are input into a pre-trained police element parsing network for joint reasoning to obtain an element parsing map;

[0009] The case elements are collected and a report is generated by analyzing the element analysis map to obtain a key case element distribution map and a preliminary case report. The key case element distribution map is used to show the spatial distribution of the core elements of the case, and the preliminary case report is used to support case analysis and decision-making.

[0010] Preferably, the training method for the police element parsing network includes:

[0011] Based on a pre-set joint training mechanism and a constructed police sample library, the police element parsing network is jointly trained in stages to obtain a trained police element parsing network. The police sample library includes image samples, spatiotemporal feature samples, and text semantic samples.

[0012] The joint training mechanism includes one or more of the following: single joint training mechanism, alternating training and parameter fixing mechanism, and alternating training and independent mechanism; the alternating training and parameter fixing mechanism includes an overall joint training unit and a model adaptation training unit; the alternating training and independent mechanism includes one or more of the following: feature encoding collaborative unit, inference decoding collaborative unit, and overall optimization collaborative unit.

[0013] Preferably, based on a preset joint training mechanism and a constructed police sample database, the police element parsing network is subjected to phased joint training to obtain a trained police element parsing network, including:

[0014] The constructed police sample database is input into the police element parsing network for joint analysis, and the parsing mask training results and element association training vectors output by the police element parsing network are obtained.

[0015] Based on the training results of the parsing mask, the training vectors associated with elements, and the preset element consistency judgment criteria, the joint training loss value of the police element parsing network is determined.

[0016] Based on the joint training loss value and the preset joint training mechanism, determine whether the police element parsing network has reached the preset training completion standard;

[0017] When the police element parsing network reaches the training completion standard, it is determined that the trained police element parsing network has reached a usable state. The trained police element parsing network is used for joint parsing of police elements.

[0018] When the police element analysis network fails to meet the training completion standard, the police element analysis network is adjusted according to the preset joint training mechanism until the police element analysis network is trained.

[0019] Preferably, according to a preset joint training mechanism, joint training and adjustment operations are performed on the police element analysis network, including:

[0020] In response to the preset joint training mechanism, including the single joint training mechanism, the overall parameter gradient update operation is performed on the police element parsing network based on the joint training loss value.

[0021] In response to a preset joint training mechanism including alternating training and parameter fixing mechanisms, and when the joint training mechanism includes model adaptation training units, the police element parsing network is subjected to model parameter fixing and gradient update operations based on the joint training loss value.

[0022] In response to a preset joint training mechanism that includes alternating training and parameter fixing mechanisms, and when the joint training mechanism includes an overall joint training unit, the police element parsing network is subjected to parameter fixing and gradient update operations, excluding the parsing head, based on the joint training loss value.

[0023] In response to a preset joint training mechanism including alternating training and independent mechanisms, and the joint training mechanism including a feature encoding cooperative unit, the police element parsing network is subjected to a gradient update operation on the encoding part based on the determined encoding cooperative loss value.

[0024] In response to a preset joint training mechanism including alternating training and independent mechanisms, and the joint training mechanism including an inference-decoding collaborative unit, the police element parsing network is subjected to encoding parameter fixing and decoding gradient update operations based on the joint training loss value.

[0025] In response to a preset joint training mechanism that includes alternating training and independent mechanisms, and the joint training mechanism includes an overall optimization collaborative unit, an overall parameter gradient update operation is performed on the police element parsing network based on the joint training loss value.

[0026] Preferably, the constructed police sample database is input into the police element parsing network for joint analysis to obtain the parsing mask training results and element association training vectors output by the police element parsing network, including:

[0027] Multi-source feature fusion is performed on image samples and spatiotemporal feature samples to obtain a fused feature vector;

[0028] Based on a preset model identifier, feature concatenation and dimension unification operations are performed on text semantic samples and fused feature vectors to obtain a multi-source joint feature tensor, wherein the model identifier includes a model location identifier and a model type identifier.

[0029] The multi-source joint feature tensor is input into the multi-source coding model in the police element parsing network for processing to obtain the coding feature tensor.

[0030] Preferably, the constructed police sample database is input into the police element parsing network for joint analysis to obtain the parsing mask training results and element association training vectors output by the police element parsing network, and further includes:

[0031] The encoded feature tensor is reconstructed to obtain the reconstructed image feature tensor. The reconstructed image feature tensor is then transformed to obtain the transformation result. The transformation result is then input into the multi-source decoding model in the police element parsing network for upsampling and feature connection operations to obtain the parsing mask training result.

[0032] Based on the coding feature tensor, the target coding component is determined. Based on the connection layer in the police element parsing network and the preset mapping function, the target coding component is labeled and vector transformed to obtain the element association training vector.

[0033] Preferably, the joint training loss value of the police element parsing network is determined based on the training results of the parsing mask, the element association training vector, and the preset element consistency judgment criterion, including:

[0034] The encoding collaborative loss value is determined based on the element association training vector and the set cross-entropy loss; the decoding collaborative loss value is determined based on the parsing mask training results and the set multi-source overlap loss calculation method; and the parsing collaborative loss value is determined based on the parsing mask training results and the set multi-source focusing loss.

[0035] Based on the determined weight combination, encoding collaborative loss value, decoding collaborative loss value, and parsing collaborative loss value, the joint training loss value of the police element parsing network is determined.

[0036] Preferably, a structured semantic parsing operation is performed on the full case file text of the target case to obtain semantic vectors of case elements, including:

[0037] Extract effective content from the full case file of the target case to obtain effective text content;

[0038] The valid text content is input into a pre-trained text parsing model for processing to obtain the text parsing result;

[0039] The text parsing results are input into a pre-trained semantic vector generation model for processing to obtain semantic vectors of case elements.

[0040] Preferably, the effective content is extracted from the full case file of the target case to obtain the effective text content, including:

[0041] The entire case file text of the target case is processed by word segmentation to obtain word segmentation results, wherein the word segmentation results include one or more word segmentation fragments;

[0042] Based on the word segmentation results and the preset keyword extraction rules, determine the keyword extraction results;

[0043] Based on the keyword extraction results and word segmentation results, semantic relevance analysis is performed to obtain the relevance score results for each word segmentation fragment.

[0044] Based on each word segment and its corresponding relevance score, the effective text content is determined.

[0045] A task processing system based on multi-agent cooperation, applicable to the aforementioned task processing method based on multi-agent cooperation, includes:

[0046] The semantic parsing module is used to perform structured semantic parsing on the full case file text of the target case to obtain semantic vectors of case elements;

[0047] The image analysis module is used to perform multi-scale image analysis on the panoramic image data of the target case scene to obtain the basic image feature tensor and the local detail feature tensor. Spatiotemporal correlation feature extraction is performed on the local detail feature tensor to obtain the spatiotemporal correlation feature results.

[0048] The joint reasoning module is used to input the semantic vectors of case elements, the basic image feature tensors, and the spatiotemporal correlation feature results into the pre-trained police element parsing network for joint reasoning to obtain the element parsing map;

[0049] The case analysis module is used to collect case elements and generate reports from the element analysis map, resulting in a key case element distribution map and a preliminary case report. The key case element distribution map is used to display the spatial distribution of the core elements of the case, and the preliminary case report is used to support case analysis and decision-making.

[0050] Compared with the prior art, the beneficial effects of the present invention are:

[0051] (1) By comprehensively processing case file text, image data and spatiotemporal features, this invention can effectively integrate multimodal data from different sources, enabling police officers to obtain more comprehensive and accurate information when handling cases. This data fusion enhances the depth and breadth of case analysis, reduces misunderstanding caused by information silos, and extracts key information from case file text through structured semantic parsing operations, so that case elements can be presented clearly and systematically. This not only improves the efficiency of case analysis, but also enhances the decision-making ability of police officers.

[0052] (2) By performing multi-scale analysis on panoramic image data of the scene and combining it with the extraction of spatiotemporal correlation features, this invention can help police officers extract potential key information from complex images. This operation is of great value for case scene reconstruction, dynamic tracking and scene analysis. Furthermore, by performing joint reasoning through a pre-trained police element analysis network, multiple data sources can be combined for analysis, thereby automatically generating element analysis maps. This process reduces manual intervention, improves the automation level of case analysis, and makes the processing speed faster and the decision-making more scientific.

[0053] (3) By generating a distribution map of key case elements, this invention can clearly show the spatial distribution of the core elements in the case, helping police officers to identify key locations, activity trajectories and key moments in the case. This visualization result enhances the ability to analyze the spatial and temporal background of the case. Furthermore, the joint training mechanism enables the model to adapt to various training methods, improving the robustness and adaptability of the network in real-world scenarios. Whenever the police element analysis network fails to meet the training standards, it can be automatically adjusted and optimized to ensure the training effect, thereby improving the stability and reliability of the analysis network. Attached Figure Description

[0054] Figure 1 This is a schematic flowchart of the overall method in one embodiment of the present invention;

[0055] Figure 2 This is a schematic diagram of the overall system architecture in one embodiment of the present invention.

[0056] In the diagram: 1. Semantic parsing module; 2. Image analysis module; 3. Joint reasoning module; 4. Case analysis module. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] Example 1, please refer to Figure 1 This invention provides a technical solution: a task processing method based on multi-agent collaboration, comprising:

[0059] S1. Perform structured semantic parsing on the full case file text of the target case to obtain the semantic vector of case elements;

[0060] S2. Perform multi-scale image analysis on the panoramic image data of the target case to obtain the basic image feature tensor and the local detail feature tensor. Perform spatiotemporal correlation feature extraction on the local detail feature tensor to obtain the spatiotemporal correlation feature results.

[0061] S3. Input the semantic vector of case elements, the basic image feature tensor, and the spatiotemporal correlation feature results into the pre-trained police element parsing network for joint reasoning to obtain the element parsing map;

[0062] S4. Perform case element aggregation and report generation operations on the element analysis map to obtain the key case element distribution map and the preliminary case report. The key case element distribution map is used to show the spatial distribution of the core elements of the case, and the preliminary case report is used to support case analysis and decision-making.

[0063] It should be noted that the textual materials in the case are subjected to structured analysis and transformed into a form that machines can understand; Natural Language Processing (NLP) technology is used to extract key information from the case and represent this information as semantic vectors.

[0064] By analyzing image data, key visual information of the crime scene can be extracted; multi-scale image analysis can analyze images at different resolutions and levels of detail to extract basic features and local details; then, spatiotemporal correlation feature extraction is used to analyze the temporal and spatial changes in the images to help understand the process of the crime.

[0065] By using machine learning models for joint reasoning, textual and visual information are combined to obtain a case element analysis map. The pre-trained police element analysis network can use case text and image information to comprehensively infer various details of the case and generate a map containing various case elements. The map may contain different nodes and connections, representing the core elements in the case and their interrelationships.

[0066] The core elements of a case are collected and a report is generated for case analysis and decision support; key information extracted from the map is organized to generate a case distribution map and report; the key case element distribution map shows the spatial distribution of the core elements of the case, while the preliminary case report summarizes the key details of the case to support subsequent decision-making.

[0067] In an optional embodiment, the training method for the police element parsing network includes:

[0068] Based on a pre-defined joint training mechanism and a constructed police sample database, the police element parsing network is jointly trained in stages to obtain a trained police element parsing network. The police sample database includes image samples, spatiotemporal feature samples, and text semantic samples.

[0069] Among them, the joint training mechanism includes one or more of the following: single joint training mechanism, alternating training and parameter fixing mechanism, and alternating training and independent mechanism; the alternating training and parameter fixing mechanism includes the overall joint training unit and the model adaptation training unit; the alternating training and independent mechanism includes one or more of the following: feature encoding collaborative unit, inference decoding collaborative unit, and overall optimization collaborative unit.

[0070] It should be noted that the training method of the police element analysis network is based on a pre-defined joint training mechanism. The purpose is to integrate training data from different sources to form a powerful multimodal model for accurately analyzing case elements. Image samples include surveillance videos and on-site photos, which provide visual information about the case. Spatiotemporal feature samples, such as spatiotemporal data (dynamic changes in time and space) in surveillance videos, help the network understand the time and location of the incident and the relationships between events. Textual semantic samples provide background information and textual descriptions of the case. Through joint training with these different types of samples, the police element analysis network can better combine visual, spatiotemporal, and textual information for reasoning and analysis. The joint training mechanism is the core of the model training process. Different training strategies are used to optimize the model so that it can simultaneously process and learn information from multiple modalities (images, spatiotemporal features, and text).

[0071] Image, spatiotemporal features, and text data are input into the network together for a single training run; the model learns these three types of information simultaneously during a training process.

[0072] Alternating training mechanisms refer to processing data from different modalities in stages during model training; parameter fixing mechanisms refer to locking (fixing) the parameters of certain layers of the network at certain stages, training only other parts; overall joint training units: at certain stages, all modalities (images, spatiotemporal features, text) are trained together, but the parameters of certain layers or parts of the network remain fixed; this can reduce training complexity and avoid overfitting; model-adaptive training units: different parts of the network are adjusted according to task requirements; for example, the parameters of the image analysis module can be fixed, focusing on training the spatiotemporal feature part, or vice versa; for example: assuming that a certain stage requires strengthening the processing capability of image data, the parameters of the image processing module can be fixed, training only the spatiotemporal feature and text parts to adapt to the needs of different modalities.

[0073] Alternating training and independent mechanisms separate data processing for each modality and optimize it in independent training units, while simultaneously optimizing the final network performance through collaborative work. Feature encoding collaborative units: Feature encoders for different modalities (such as image encoders, text encoders, etc.) are trained and optimized separately, then collaborate at a certain stage to ensure that features are effectively combined in the overall model. Inference decoding collaborative units: These are collaborative units in the inference stage, where the model combines information from all modalities to reason about the case; this part is usually performed alternately in multiple training stages. Overall optimization collaborative units: These combine information from all modalities to perform final optimization adjustments, ensuring that multimodal features are fully utilized during inference and improving the overall performance of the model. For example: In the early stages of training, the network may independently train the image encoder, spatiotemporal feature encoder, and text encoder, gradually allowing them to learn the features of their respective modalities; later, the network will enter the inference stage, where the model combines information from all modalities to make inferences.

[0074] In an optional embodiment, based on a preset joint training mechanism and a constructed police sample database, the police element parsing network is subjected to phased joint training to obtain a trained police element parsing network, including:

[0075] The constructed police sample database is input into the police element parsing network for joint analysis, and the parsing mask training results and element association training vectors output by the police element parsing network are obtained.

[0076] Based on the training results of the parsing mask, the training vectors associated with elements, and the preset element consistency judgment criteria, the joint training loss value of the police element parsing network is determined.

[0077] Based on the joint training loss value and the preset joint training mechanism, determine whether the police element parsing network has reached the preset training completion standard;

[0078] When the police element parsing network reaches the training completion standard, it is determined that the trained police element parsing network has reached a usable state. The trained police element parsing network is used for joint parsing of police elements.

[0079] When the police element analysis network fails to meet the training completion standard, the police element analysis network is adjusted according to the preset joint training mechanism until the police element analysis network is trained.

[0080] It should be noted that the police sample database includes image samples, spatiotemporal feature samples, and text semantic samples. During training, these samples from the police sample database are input into the police element parsing network and trained through joint analysis. Parsing mask training result: The network generates a "parsing mask", which is the analysis result of the input data and represents the model's prediction of each police element. Element association training vector: This is another output generated by the network, representing the association between different police elements.

[0081] The loss value is used to measure the difference between the model output and the actual label. Specifically: the parsed mask training result refers to the model's prediction results for different police elements; the element association training vector represents the relationship between various police elements and, combined with spatiotemporal data, establishes the spatiotemporal association between the target person and the event; if the actual situation is inconsistent with the model's prediction, the error of the element association vector will also affect the loss calculation; the element consistency judgment criterion is a standard used to judge whether the parsed mask and element association vector output by the model conform to the actual situation; for example, if the identity of a target person is predicted to be "Li Si", but the person in the video does not match the appearance characteristics of the actual target person, it is judged to be inconsistent; through this information, the network will calculate a joint training loss value, which reflects the difference between the network output and the actual label.

[0082] Joint training loss: This is calculated by the difference between the parsed mask training result and the feature association training vector and the actual result. For example, assuming the difference between the parsed mask output by the network and the actual mask is 0.2, and the error of the feature association training vector is 0.1, the final loss value may be 0.3. Training completion standard: This refers to the preset loss value threshold. If the loss value during the training process is lower than a certain standard, it means that the network training effect is good and it can effectively parse police elements. In some cases, the loss value may need to be lower than 0.1 to be considered as training successful. The specific standard will be set according to the requirements of the task.

[0083] Training completion criteria met: If the loss value output by the network is less than or equal to the preset standard, it means that the network has been able to effectively analyze police elements and the training is complete. At this time, the model can be used to process new police data for element analysis. Training completion criteria not met: If the loss value is still large, it means that the training effect of the network is not ideal and cannot effectively process police data. In this case, it is necessary to further adjust the network parameters or training strategy and continue to train the network until the preset training completion criteria are met.

[0084] When training fails to meet the preset standards, joint training adjustments are necessary. This typically involves the following steps: adjusting the training strategy: this may involve changing the training method, such as using different joint training mechanisms (e.g., alternating training versus independent training) to retrain the network; adjusting model parameters: adjusting the network's weights and biases to optimize the training results; and expanding training data: adding more samples to train the model, especially those weaker parts, such as adding more image samples, spatiotemporal feature samples, or text data. When the network completes training and meets the preset standards, the trained police element analysis network is considered to have reached a "usable state." At this point, the model can be applied to practical police element analysis tasks.

[0085] In an optional embodiment, according to a preset joint training mechanism, a joint training adjustment operation is performed on the police element parsing network, including:

[0086] In response to the preset joint training mechanism, including the single joint training mechanism, the overall parameter gradient update operation is performed on the police element parsing network based on the joint training loss value.

[0087] In response to the preset joint training mechanism, which includes alternating training and parameter fixing mechanisms, and when the joint training mechanism includes model adaptation training units, the police element parsing network is subjected to model parameter fixing and gradient update operations based on the joint training loss value.

[0088] In response to the preset joint training mechanism, which includes alternating training and parameter fixing mechanisms, and when the joint training mechanism includes the overall joint training unit, the police element parsing network is subjected to parameter fixing and gradient update operations except for the parsing head, based on the joint training loss value.

[0089] When the preset joint training mechanism includes alternating training and independent mechanism and the joint training mechanism includes feature encoding cooperative unit, the gradient update operation of the encoding part of the police element parsing network is performed based on the determined encoding cooperative loss value.

[0090] When the preset joint training mechanism includes alternating training and independent mechanism and the joint training mechanism includes inference-decoding collaborative unit, the police element parsing network is subjected to encoding parameter fixing and decoding gradient update operation based on the joint training loss value.

[0091] When the preset joint training mechanism includes alternating training and independent mechanism and the joint training mechanism includes overall optimization collaborative unit, the overall parameter gradient update operation is performed on the police element parsing network based on the joint training loss value.

[0092] It should be noted that when using the single joint training mechanism, all network parameters will be updated with gradients based on the joint training loss value. Under this mechanism, the police element parsing network will be trained once for the entire network (including all layers), and all parameters will be updated based on the joint training loss value (such as parsing error and element association error). If the parsing result given by the network after the first training is significantly different from the actual result (e.g., identity recognition error), then all parameters of the network (e.g., convolutional layers, fully connected layers, etc.) will be updated according to the loss value to optimize its parsing ability.

[0093] Alternating training and parameter fixation mechanisms, when used in a joint training mechanism, involve alternating updates to different network parts. The network includes a model adaptation training unit (MAU) that updates or fixes the overall parameters using the joint training loss value. Under this mechanism, parameters of certain layers (e.g., convolutional layers) are fixed, while others (e.g., classification heads) are updated. The MAU updates the parameters of the adapted network to fit different data sources. For example, consider training a network combining video analysis and spatiotemporal data. In the initial training phase, the image recognition part (convolutional layers) may be trained to sufficient accuracy. Therefore, the parameters of these layers are fixed, and only the parameters of the time series analysis part (e.g., the Long Short-Term Memory (LSTM) part) are updated to optimize the network's performance in temporal information processing.

[0094] Alternating training and parameter fixing mechanisms, with a holistic joint training unit, involve a single holistic joint training unit where all network parameters, except for the parser head, are fixed, with only gradient updates performed on specific parts. In this mechanism, the parser head (e.g., the network's output layer) is trained, while the parameters of other parts remain fixed, with adjustments made only to specific network components. Example: Suppose a video analysis model is being trained to identify individuals from shopping mall surveillance footage. Since the network is already capable of accurately extracting features from video frames, only the parser head is adjusted, without updating other layers (such as convolutional layers), to improve the accuracy of the final identity recognition.

[0095] Alternating training and independence mechanism, feature encoding co-unit: In this mechanism, the joint training mechanism includes a feature encoding co-unit, where gradient updates are performed on the encoding part of the network based on the encoding co-loss value. This strategy emphasizes the optimization of feature encoding and may use some additional supervision signals to help the encoding part (such as the feature extraction part) learn more effectively. Gradient updates only occur in the encoding part. Example: If the network task is to simultaneously identify people and their actions in a surveillance video, then the encoding part (e.g., the feature extraction part) may need more optimization to capture details related to the people's actions. In this case, using the alternating training and independence mechanism, the parameters of the encoding part are updated separately, while the decoding part (e.g., the final action recognition classifier) ​​remains unchanged.

[0096] Alternating training and independent mechanisms, inference-decoding co-operational units: Under this mechanism, joint training includes an inference-decoding co-operational unit. Based on the joint training loss value, the parameters of the encoding part are fixed, while the parameters of the decoding part (i.e., the inference part) are updated using gradients. This mechanism primarily focuses on optimizing the decoding part, especially mapping the high-dimensional feature space back to the lower-dimensional label space during inference. The decoding part is updated, while the encoding part (such as the feature extraction part of an image) is not. Example: Suppose the goal is to infer the movement trajectory of a target person from surveillance video. The feature extraction part is already well-trained, so its parameters are fixed. Next, the decoding part (e.g., a temporal inference network) is optimized to accurately predict the target person's path from the extracted features.

[0097] Alternating training and independent mechanisms, combined with a holistic optimization collaborative unit, combine alternating training and independent mechanisms while incorporating a holistic optimization collaborative unit. Under this mechanism, all network parameters undergo gradient updates, and overall optimization is performed based on the joint training loss value. This mechanism allows optimization of all parts of the entire network (encoding, inference, decoding, etc.) to improve the overall model performance. Example: For a complex police element analysis task, suppose the network needs to simultaneously perform identity recognition, behavior prediction, and spatiotemporal trajectory analysis. To ensure that all tasks are optimized, the network will simultaneously train and update all parts (feature encoding, behavior recognition, spatiotemporal prediction, etc.) to achieve the final comprehensive optimization.

[0098] In an optional embodiment, the constructed police sample database is input into a police element parsing network for joint analysis to obtain the parsing mask training results and element association training vectors output by the police element parsing network, including:

[0099] Multi-source feature fusion is performed on image samples and spatiotemporal feature samples to obtain a fused feature vector;

[0100] Based on the preset model identifier, feature concatenation and dimension unification operations are performed on the text semantic samples and fused feature vectors to obtain a multi-source joint feature tensor. The model identifier includes a model location identifier and a model type identifier.

[0101] The multi-source joint feature tensor is input into the multi-source coding model in the police element parsing network for processing to obtain the coding feature tensor.

[0102] It's important to note that fusing image data (such as video frames or images) with spatiotemporal feature data (such as timestamps, spatial locations, etc.) allows for the generation of a fused feature vector from two different types of input features. Image samples may contain image information about a specific region or person, while spatiotemporal feature samples include information such as the time and location where the image occurred. By using methods (such as weighted averaging, stitching, etc.), image samples and spatiotemporal feature samples are combined into a unified feature vector, thus capturing both image and spatiotemporal information simultaneously. For example, suppose we are processing a task to track people in a surveillance video. Each frame of image data contains image information about the person, while the spatiotemporal feature data contains the timestamp of that frame and the spatial location of the scene captured in that frame. By fusing these two types of features, the resulting fused feature vector contains the image features of the person and the spatiotemporal information of the time and location of the image.

[0103] The text semantic sample and the fused feature vector obtained in the previous step are concatenated. Then, the dimensions need to be unified using a preset model identifier to ensure that all features have a uniform dimension, adapting to the processing of downstream models. The text semantic sample and the fused feature vector are concatenated together to form a composite feature containing textual, image, and spatiotemporal information. The dimensions of different features are adjusted according to the model identifier to ensure that the final concatenated feature tensor has a uniform dimension. For example, if the text feature dimension is 200, while the image and spatiotemporal features have a dimension of 500, they may need to be adjusted to the same dimension in some way (e.g., through linear transformation, pooling operations, etc.). The text semantic sample may include a brief case description, while the fused feature vector may contain the image features of the person and the time and location of entering the mall. These two parts of features are concatenated into a composite feature vector, and then the dimensions are adjusted using a preset model identifier so that they can be passed as a uniform input to the subsequent network model.

[0104] The multi-source joint feature tensor, after concatenation and dimensional unification, is input into the multi-source coding model in the police element parsing network for processing, resulting in an encoded feature tensor. The multi-source coding model combines features from different sources (such as images, text, and spatiotemporal information) and further processes these features using specific coding methods (such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), or Transformer models) to generate deeper encoded features. This encoded feature tensor retains information from different sources and is optimized through deep networks. These multi-source features are then input into the multi-source coding model, which may use convolutional networks to process image features, RNNs or Transformers to process text information, and combine spatiotemporal information for temporal analysis. Finally, the model outputs an encoded feature tensor that contains a comprehensive analysis of the individuals.

[0105] In an optional embodiment, the constructed police sample database is input into a police element parsing network for joint analysis to obtain the parsing mask training results and element association training vectors output by the police element parsing network, and the method further includes:

[0106] The encoded feature tensor is reconstructed to obtain the reconstructed image feature tensor. The reconstructed image feature tensor is then transformed to obtain the transformation result. The transformation result is then input into the multi-source decoding model in the police element parsing network for upsampling and feature connection operations to obtain the parsing mask training result.

[0107] Based on the coding feature tensor, the target coding component is determined. Based on the connection layer in the police element parsing network and the preset mapping function, the target coding component is labeled and vector transformed to obtain the element association training vector.

[0108] It should be noted that the goal of feature reconstruction is to restore the encoded feature tensor back to the form of image data, or at least convert it into a tensor related to image features. This process typically requires inferring certain key features of the image data based on the encoded information. By using backpropagation and neural network architectures such as autoencoders and variational autoencoders (VAEs), or by using deep learning models to decode the encoded feature tensor, the feature representation of the image or video is reconstructed. For example, suppose the image features and spatiotemporal information of a video frame have been encoded into a high-dimensional feature tensor. Next, the feature reconstruction operation "decodes" this high-dimensional tensor back into a more readable image feature, which may be about the image features of a certain area in the surveillance video. This process infers elements related to the image features from the encoded features, reflecting the structure and patterns of the image data.

[0109] Dimensionality transformation operations are typically performed to adjust the dimensions of feature tensors to adapt to the requirements of subsequent processing steps or to reduce computational complexity. Through transformation, feature tensors can be made suitable for input into different types of network modules or to optimize model performance. Common dimensionality transformation operations include feature dimension compression (such as through pooling operations or principal component analysis (PCA)) or by adding additional dimensions (e.g., using convolutional layers, fully connected layers, etc.). These operations are usually performed to improve efficiency in specific computational environments, reduce unnecessary feature dimensions, or extract more important features. For example, suppose an image of a person's region in a surveillance video has been reconstructed; the image dimension may be very high (e.g., 2048×2048 pixels). To reduce the computational burden and adapt to subsequent processing, the image may be dimensionality-reduced, converted into a lower-resolution feature representation (e.g., 256×256), or convolutional operations may be used to extract feature maps containing key information.

[0110] Upsampling and feature concatenation are common operations in decoding networks, aiming to restore processed low-dimensional features to the original data dimension or to integrate multiple feature information together. Upsampling is typically used to restore low-resolution feature maps to a high-resolution form, while feature concatenation merges features from multiple different sources to form a more complete representation. Multi-source decoding models typically use upsampling (e.g., using deconvolutional layers) to restore low-dimensional features to higher dimensions, and use feature concatenation to concatenate or weightedly merge multiple feature tensors to form the final output. In police element parsing, this process is often used to generate a "parsing mask" that represents specific police-related elements. If the image resolution is reduced to 256×256, the decoding network can use upsampling to restore it to a higher resolution (e.g., 1024×1024), and use feature concatenation to combine features from different sources (such as spatiotemporal information and image information) to generate a mask.

[0111] Based on the encoded feature tensor, target encoded components are determined. Using the connection layers in the police element analysis network and a pre-defined mapping function, feature annotation and vector transformation operations are performed on the target encoded components to obtain element-related training vectors. Target encoded components refer to the feature parts selected from the encoded feature tensor that are crucial to the analysis of police elements. Based on these target encoded components, feature annotation and vector transformation can be performed through connection layers and mapping functions. Fully connected layers or other types of layers (such as attention mechanisms) in the network are used to determine which encoded components are most relevant to police elements. Then, a pre-defined mapping function (e.g., linear transformation, nonlinear transformation) is used to convert these components into specific feature labels, and a training vector is obtained through vector transformation. This training vector can be used for subsequent training and model optimization.

[0112] In an optional embodiment, the joint training loss value of the police element parsing network is determined based on the parsing mask training results, the element association training vectors, and a preset element consistency judgment criterion, including:

[0113] The encoding collaborative loss value is determined based on the element association training vector and the set cross-entropy loss; the decoding collaborative loss value is determined based on the parsing mask training results and the set multi-source overlap loss calculation method; and the parsing collaborative loss value is determined based on the parsing mask training results and the set multi-source focusing loss.

[0114] Based on the determined weight combination, encoding collaborative loss value, decoding collaborative loss value, and parsing collaborative loss value, the joint training loss value of the police element parsing network is determined.

[0115] It should be noted that the calculation of the encoding co-encoding loss value is based on the feature association training vector and the cross-entropy loss. Cross-entropy loss is usually used in classification problems to measure the difference between the true label and the model's prediction. Here, cross-entropy loss is used to measure the difference between the feature association vector generated by the model during the encoding stage and the true label. Given a target feature, the network outputs a vector representing the probability or class distribution of that feature. By comparing it with the true label, cross-entropy can be used to calculate the error and then adjust the network weights to reduce this error.

[0116] The decoding collaborative loss value is determined based on the parsing mask training results and the multi-source overlap loss calculation method. The multi-source overlap loss focuses on the degree of overlap between different feature sources in the final output, aiming to optimize the features generated in the decoding stage so that data from different sources can be organically combined without information redundancy or loss. During the parsing mask training process, the model generates predictions based on different sources (e.g., image data, spatiotemporal information, behavioral patterns, etc.). The multi-source overlap loss measures the overlapping part between different sources and requires the decoding model to minimize overlap as much as possible, maintaining the independence and complementarity of information from different sources. For example, in a surveillance scenario, suppose there is image data and time series data from a camera. During the decoding process, the model needs to fuse features from these different sources. However, if the information from different sources overlaps or repeats excessively, it may lead to unnecessary information redundancy, affecting recognition accuracy. By calculating the multi-source overlap loss, the network is guided to avoid this redundancy, allowing the information from each source to play its unique role in the final parsing result.

[0117] The parsing collaborative loss value is determined based on the parsing mask training results and the multi-source focusing loss. The multi-source focusing loss aims to improve the accuracy of the decoding model in the task of parsing police elements by focusing on the most critical information sources. By setting a multi-source focusing loss, the model will learn to prioritize certain information sources (e.g., certain image features or behavioral patterns) according to different task requirements, thereby optimizing the parsing process and avoiding over-reliance on a single data source. Ultimately, the generation of the parsing mask will be more accurate and can accurately label police elements.

[0118] The final joint training loss is calculated based on the three previous loss values ​​(encoding collaborative loss, decoding collaborative loss, and parsing collaborative loss) and their corresponding weight combinations. These weight combinations reflect the importance of each loss in network training. By weighting different losses, the model can be guided to find a balance among multiple objectives, optimizing the entire police element parsing network. During training, the weights of each loss function can be adjusted according to task requirements. For example, if more emphasis is placed on the accuracy of the encoding stage, the weight of the encoding collaborative loss can be increased; if the accuracy of the decoding stage is more important, the weight of the decoding collaborative loss can be increased. Finally, the weighted sum of all these loss values ​​yields the joint training loss, which is used for backpropagation to adjust the network parameters.

[0119] In an optional embodiment, a structured semantic parsing operation is performed on the full case file text of the target case to obtain semantic vectors of case elements, including:

[0120] Extract effective content from the full case file of the target case to obtain effective text content;

[0121] The valid text content is input into a pre-trained text parsing model for processing to obtain the text parsing result;

[0122] The text parsing results are input into a pre-trained semantic vector generation model for processing to obtain semantic vectors of case elements.

[0123] It should be noted that extracting useful and key information from the full case file text is crucial. Case files may contain a large amount of irrelevant information or redundant data, so an effective content extraction mechanism is needed to filter out the parts useful for case analysis. Effective content extraction typically includes the following steps: text cleaning: removing useless text content, such as headers, footers, formatting information, irrelevant blank parts, etc.; information extraction: extracting key entities from the text, such as case number, case type, target personnel, date, etc., through methods such as keywords and non-representative entity recognition (NER).

[0124] The extracted valid text content is input into a pre-trained text parsing model for further in-depth processing. The purpose of the text parsing model is to understand the text through natural language processing techniques and extract deeper structured information. The text parsing model usually uses pre-trained language models based on deep learning (such as BERT, GPT, etc.) for natural language understanding. The goal is to parse out the grammatical structure, contextual relationships, event sequence, and other content in the text. At this stage, the model will not only simply identify information, but will also be able to understand the relationships between information.

[0125] The output of the text parsing model will be used as input to a pre-trained semantic vector generation model. The goal of the semantic vector generation model is to transform the case text parsing results into semantic vectors that can be used for calculation, comparison, and analysis. The semantic vector generation model is usually based on natural language processing techniques (such as Word2Vec, BERT, Sentence-BERT, etc.) to transform information in the text into fixed-length vector representations. These vectors can be compared between different texts to measure their semantic similarity. The vector represents the semantic content of the text and can be used for further classification, clustering, or retrieval tasks.

[0126] In an optional embodiment, a valid content extraction operation is performed on the full case file text of the target case to obtain valid text content, including:

[0127] The entire case file text of the target case is processed by word segmentation to obtain word segmentation results, which include one or more word segmentation fragments;

[0128] Based on the word segmentation results and the preset keyword extraction rules, determine the keyword extraction results;

[0129] Based on the keyword extraction results and word segmentation results, semantic relevance analysis is performed to obtain the relevance score results for each word segmentation fragment.

[0130] Based on each word segment and its corresponding relevance score, the effective text content is determined.

[0131] It should be noted that word segmentation is the process of breaking down a continuous piece of text into independent, meaningful word segments. Chinese text does not have explicit spaces between words, so word segmentation tools are needed to divide the text into reasonable words or phrases (called word segments). When performing word segmentation, some Chinese natural language processing (NLP) tools or libraries are usually used, such as jieba segmenter, HanLP, etc., to segment the text. The granularity of word segmentation can be adjusted according to specific needs, such as segmenting into single characters, words, or longer phrases.

[0132] Keyword extraction rules are used to filter out important keywords in a text by setting a series of predefined rules. These rules can be based on grammar (such as the extraction of nouns and verbs), contextual information, or pre-defined domain vocabulary. Based on the word segmentation results and the preset keyword rules, key information related to the case is extracted. The rules may include: extracting specific types of words; extracting proper nouns or other phrases with clear meaning.

[0133] Semantic relevance analysis refers to using an algorithm to evaluate the semantic relevance of each segmented word to keywords, based on word segmentation and keyword extraction. The goal is to identify which words or segments in the text are most relevant to key elements of a case. Typically, semantic relevance analysis can be based on the following methods: TF-IDF model: measures the importance of a word in the text; cosine similarity: calculates the similarity between segmented words and keywords; deep learning models such as BERT or Word2Vec: calculate semantic vectors and determine relevance by calculating the vector distance between words.

[0134] Based on the semantic relevance scoring results above, we can determine which word segments are valid text content. Valid content refers to information that is highly relevant to the core elements of the case. By setting a threshold (for example, segments with a relevance score greater than 0.8 are considered valid content), we can filter out these key texts. Based on the relevance score of each word segment, we select segments with higher scores and consider them to be valid information in the case. This valid information will constitute the key information of the case and can be further used for case analysis, reasoning, etc.

[0135] Example 2, please refer to Figure 2 This invention provides a technical solution: a task processing system based on multi-agent collaboration, applicable to the aforementioned task processing method based on multi-agent collaboration, comprising:

[0136] Semantic parsing module 1 is used to perform structured semantic parsing on the full case file text of the target case to obtain semantic vectors of case elements;

[0137] Image analysis module 2 is used to perform multi-scale image analysis on the panoramic image data of the target case scene to obtain the basic image feature tensor and the local detail feature tensor. Spatiotemporal correlation feature extraction is performed on the local detail feature tensor to obtain the spatiotemporal correlation feature results.

[0138] Joint reasoning module 3 is used to input the semantic vector of case elements, the basic image feature tensor and the spatiotemporal correlation feature results into the pre-trained police element parsing network for joint reasoning to obtain the element parsing map;

[0139] The case analysis module 4 is used to collect case elements and generate reports from the element analysis map, resulting in a key case element distribution map and a preliminary case report. The key case element distribution map is used to display the spatial distribution of the core elements of the case, and the preliminary case report is used to support case analysis and decision-making.

[0140] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited thereto. Various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention.

Claims

1. A task processing method based on multi-agent cooperation, characterized in that, include: A structured semantic parsing operation is performed on the full case file text of the target case to obtain the semantic vector of case elements; Multi-scale image analysis is performed on the panoramic image data of the target case to obtain the basic image feature tensor and the local detail feature tensor. Spatiotemporal correlation feature extraction is performed on the local detail feature tensor to obtain the spatiotemporal correlation feature results. The semantic vectors of case elements, the basic image feature tensors, and the spatiotemporal correlation feature results are input into a pre-trained police element parsing network for joint reasoning to obtain an element parsing map; The case elements are collected and a report is generated by performing case element analysis on the element analysis map to obtain a key case element distribution map and a preliminary case report. The key case element distribution map is used to show the spatial distribution of the core elements of the case, and the preliminary case report is used to support case analysis and decision-making. The training method for the police element parsing network includes: Based on the pre-set joint training mechanism and the constructed police sample database, the police element analysis network is jointly trained in stages to obtain the trained police element analysis network. Accordingly, based on the pre-set joint training mechanism and the constructed police sample database, the police element parsing network is subjected to phased joint training to obtain the trained police element parsing network, including: The constructed police sample database is input into the police element parsing network for joint analysis, and the parsing mask training results and element association training vectors output by the police element parsing network are obtained. Accordingly, the constructed police sample database is input into the police element parsing network for joint analysis, yielding the parsing mask training results and element association training vectors output by the police element parsing network, including: Multi-source feature fusion is performed on image samples and spatiotemporal feature samples to obtain a fused feature vector; Based on a preset model identifier, feature concatenation and dimension unification operations are performed on text semantic samples and fused feature vectors to obtain a multi-source joint feature tensor, wherein the model identifier includes a model location identifier and a model type identifier. The multi-source joint feature tensor is input into the multi-source coding model in the police element parsing network for processing to obtain the coding feature tensor. This includes inputting the constructed police sample database into the police element parsing network for joint analysis, obtaining the parsing mask training results and element association training vectors output by the police element parsing network, and also including: The encoded feature tensor is reconstructed to obtain the reconstructed image feature tensor. The reconstructed image feature tensor is then transformed to obtain the transformation result. The transformation result is then input into the multi-source decoding model in the police element parsing network for upsampling and feature connection operations to obtain the parsing mask training result. Based on the coding feature tensor, the target coding component is determined. Based on the connection layer in the police element parsing network and the preset mapping function, the target coding component is labeled and vector transformed to obtain the element association training vector.

2. The task processing method based on multi-agent cooperation according to claim 1, characterized in that, The police sample library includes image samples, spatiotemporal feature samples, and text semantic samples; The joint training mechanism includes one or more of the following: single joint training mechanism, alternating training and parameter fixing mechanism, and alternating training and independent mechanism; the alternating training and parameter fixing mechanism includes an overall joint training unit and a model adaptation training unit; the alternating training and independent mechanism includes one or more of the following: feature encoding collaborative unit, inference decoding collaborative unit, and overall optimization collaborative unit.

3. The task processing method based on multi-agent cooperation according to claim 2, characterized in that, Based on a pre-defined joint training mechanism and a constructed police sample database, the police element parsing network is subjected to phased joint training to obtain a trained police element parsing network, which also includes: Based on the training results of the parsing mask, the training vectors associated with elements, and the preset element consistency judgment criteria, the joint training loss value of the police element parsing network is determined. Based on the joint training loss value and the preset joint training mechanism, determine whether the police element parsing network has reached the preset training completion standard; When the police element parsing network reaches the training completion standard, it is determined that the trained police element parsing network has reached a usable state. The trained police element parsing network is used for joint parsing of police elements. When the police element analysis network fails to meet the training completion standard, the police element analysis network is adjusted according to the preset joint training mechanism until the police element analysis network is trained.

4. The task processing method based on multi-agent cooperation according to claim 3, characterized in that, Based on the pre-set joint training mechanism, joint training and adjustment operations are performed on the police element analysis network, including: In response to the preset joint training mechanism, including the single joint training mechanism, the overall parameter gradient update operation is performed on the police element parsing network based on the joint training loss value. In response to a preset joint training mechanism including alternating training and parameter fixing mechanisms, and when the joint training mechanism includes model adaptation training units, the police element parsing network is subjected to model parameter fixing and gradient update operations based on the joint training loss value. In response to a preset joint training mechanism that includes alternating training and parameter fixing mechanisms, and when the joint training mechanism includes an overall joint training unit, the police element parsing network is subjected to parameter fixing and gradient update operations, excluding the parsing head, based on the joint training loss value. In response to a preset joint training mechanism including alternating training and independent mechanisms, and the joint training mechanism including a feature encoding cooperative unit, the police element parsing network is subjected to a gradient update operation on the encoding part based on the determined encoding cooperative loss value. In response to a preset joint training mechanism including alternating training and independent mechanisms, and the joint training mechanism including an inference-decoding collaborative unit, the police element parsing network is subjected to encoding parameter fixing and decoding gradient update operations based on the joint training loss value. In response to a preset joint training mechanism that includes alternating training and independent mechanisms, and the joint training mechanism includes an overall optimization collaborative unit, an overall parameter gradient update operation is performed on the police element parsing network based on the joint training loss value.

5. The task processing method based on multi-agent cooperation according to claim 4, characterized in that, Based on the mask training results, element association training vectors, and preset element consistency criteria, the joint training loss value of the police element parsing network is determined, including: The encoding collaborative loss value is determined based on the element association training vector and the set cross-entropy loss; the decoding collaborative loss value is determined based on the parsing mask training results and the set multi-source overlap loss calculation method; and the parsing collaborative loss value is determined based on the parsing mask training results and the set multi-source focusing loss. Based on the determined weight combination, encoding collaborative loss value, decoding collaborative loss value, and parsing collaborative loss value, the joint training loss value of the police element parsing network is determined.

6. The task processing method based on multi-agent cooperation according to claim 5, characterized in that, A structured semantic parsing operation is performed on the full case file text of the target case to obtain semantic vectors of case elements, including: Extract effective content from the full case file of the target case to obtain effective text content; The valid text content is input into a pre-trained text parsing model for processing to obtain the text parsing result; The text parsing results are input into a pre-trained semantic vector generation model for processing to obtain semantic vectors of case elements.

7. A task processing method based on multi-agent cooperation according to claim 6, characterized in that, The entire case file of the target case is subjected to effective content extraction operations to obtain effective text content, including: The entire case file text of the target case is processed by word segmentation to obtain word segmentation results, wherein the word segmentation results include one or more word segmentation fragments; Based on the word segmentation results and the preset keyword extraction rules, determine the keyword extraction results; Based on the keyword extraction results and word segmentation results, semantic relevance analysis is performed to obtain the relevance score results for each word segmentation fragment. Based on each word segment and its corresponding relevance score, the effective text content is determined.

8. A task processing system based on multi-agent cooperation, applicable to the task processing method based on multi-agent cooperation as described in any one of claims 1-7, characterized in that, include: The semantic parsing module is used to perform structured semantic parsing on the full case file text of the target case to obtain semantic vectors of case elements; The image analysis module is used to perform multi-scale image analysis on the panoramic image data of the target case scene to obtain the basic image feature tensor and the local detail feature tensor. Spatiotemporal correlation feature extraction is performed on the local detail feature tensor to obtain the spatiotemporal correlation feature results. The joint reasoning module is used to input the semantic vectors of case elements, the basic image feature tensors, and the spatiotemporal correlation feature results into the pre-trained police element parsing network for joint reasoning to obtain the element parsing map; The case analysis module is used to collect case elements and generate reports from the element analysis map, resulting in a key case element distribution map and a preliminary case report. The key case element distribution map is used to display the spatial distribution of the core elements of the case, and the preliminary case report is used to support case analysis and decision-making.

Images