Collaborative operation sensing method and device, equipment and medium

Abstract

The invention provides a collaborative operation sensing method and device, equipment and a medium, and the method comprises the steps: obtaining the trajectory data of a collaborative agent, and constructing a dynamic operation network; discretizing and compressing the dynamic job network to obtain a plurality of compressed sub-graphs; carrying out random walk and learning on each compressed sub-graph to obtain an embedded vector; the embedded vector is calculated, multi-scale features are extracted, and the type of collaborative operation is perceived. According to the invention, the cooperative agent, the trajectory data of the cooperative agent, the cooperative station and the operation scene are combined to form a dynamic operation network, and a space-time cooperative relationship is reserved. Through discrete compression, the long-time spatial trajectory is compressed into a plurality of static short-time spatial trajectories. Through random walk and learning, not only is the feature of each compressed sub-graph learned, but also the time change between different compressed sub-graphs can be captured. And finally, multi-scale features are extracted, and collaborative operation perception is carried out. The method and the device have good mobility in different operation scenes.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and deep learning, and in particular to a collaborative operation perception method, device, equipment and medium. Background Art

[0002] In the airport scenario, the ground support operations of the aircraft are key pre-flight activities to keep it in normal operation. These support operations include tasks such as cargo transfer and fuel supply, which are mainly completed by various support agents (such as refueling trucks, baggage loaders, and related personnel). Support agents collaborate effectively on the busy airport ground, moving to different workstations (such as refueling, loading and unloading, and maintenance) and performing specific tasks to ensure the normal operation of the aircraft. Perception of these support operations is crucial to analyzing and improving overall support capabilities.

[0003] The process of perceiving support operations involves analyzing the short-term and long-term collaboration patterns between support agents and various resource locations from long-term airport ground operation trajectories. However, existing perception methods perform well in perceiving collaborative behaviors in short-term trajectories, but have difficulty effectively perceiving complex support operations when dealing with longer trajectories. In addition, existing methods also lack in-depth correlation analysis of scene information such as airport resource locations, which limits the accuracy and applicability of perception results. Summary of the invention

[0004] The present invention provides a collaborative operation perception method, device, equipment and medium to solve the problem that existing perception methods are difficult to effectively perceive complex support operations when processing longer trajectories, and the existing methods also lack in-depth correlation analysis of scene information such as airport workstations, which limits the accuracy and applicability of perception results.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is to provide a collaborative work perception method, including the steps of: obtaining trajectory data of the collaborative intelligent body, and constructing a dynamic work network according to the collaborative workstation, the collaborative intelligent body and the trajectory data; wherein the dynamic work network includes multiple dynamic nodes and connecting edges; performing discretization processing and compression processing on the dynamic work network in turn to obtain multiple compressed subgraphs; performing random walks and learning on each of the compressed subgraphs to obtain an embedding vector of each of the nodes in the compressed subgraph; calculating the embedding vector, extracting multi-scale features of the embedding vector, and perceiving the type of collaborative work based on the multi-scale features.

[0006] In some embodiments, the construction of a dynamic operation network specifically includes: creating a shared spatiotemporal cellular space based on the operation scenario and the trajectory data; calculating the spatiotemporal collaborative relationship between the collaborative workstation and the collaborative agent, and between different collaborative agents based on the shared spatiotemporal cellular space; using the collaborative workstation and the collaborative agent as the nodes, and the spatiotemporal collaborative relationship as the connecting edges between the nodes, to obtain the dynamic operation network.

[0007] In some embodiments, in the step of sequentially discretizing and compressing the dynamic operation network, the discretizing includes: discretizing the dynamic operation network in time to obtain a plurality of static discrete subgraphs.

[0008] In some embodiments, in the step of sequentially performing discretization processing and compression processing on the dynamic job network, the compression processing includes: dividing the plurality of discrete subgraphs into a plurality of compression groups according to the compression time step, and merging the plurality of discrete subgraphs in each of the compression groups into one compressed subgraph.

[0009] In some embodiments, in the step of performing random walk and learning on each of the compressed subgraphs, the random walk specifically includes: for each of the compressed subgraphs, performing a target number of random walks according to a first target length; wherein each random walk obtains a random walk sequence; and using a sliding window of a second target length to slide on each of the random walk sequences to obtain multiple node pairs.

[0010] In some embodiments, in the step of performing random walk and learning on each of the compressed subgraphs, the learning specifically includes: training and learning a plurality of the node pairs through a skip-word negative sampling model to obtain a training vector for each of the node pairs; iteratively processing the training vectors of each of the node pairs in all of the compressed subgraphs to obtain the embedding vectors of all of the nodes.

[0011] In some embodiments, the calculation of the embedding vector and the extraction of multi-scale features of the embedding vector specifically include: merging all the embedding vectors into a whole and dividing them into multiple data blocks; after linearly embedding multiple data blocks, extracting the global features and local features of the data blocks; after dimensionality reduction processing of the global features and local features, extracting the multi-scale features.

[0012] The present invention also provides a collaborative work perception device, including: an acquisition construction unit, used to obtain the trajectory data of the collaborative intelligent body, and construct a dynamic work network according to the collaborative workstation, the collaborative intelligent body and the trajectory data; wherein the dynamic work network includes multiple dynamic nodes and connecting edges; a discrete compression unit, used to perform discrete processing and compression processing on the dynamic work network in turn to obtain multiple compressed sub-graphs; a walking learning unit, used to perform random walking and learning on each of the compressed sub-graphs to obtain an embedding vector of each of the nodes in the compressed sub-graph; an extraction perception unit, used to calculate the embedding vector, extract the multi-scale features of the embedding vector, and perceive the type of collaborative work based on the multi-scale features.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method when executing the computer program.

[0014] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described above are implemented.

[0015] The beneficial effects of the present invention are as follows: the present invention discloses a collaborative operation perception method, device, equipment and medium, the method comprising the steps of: obtaining the trajectory data of a collaborative agent, constructing a dynamic operation network according to the collaborative workstation, the collaborative agent and the trajectory data; discretizing and compressing the dynamic operation network in turn to obtain multiple compressed subgraphs; performing random walks and learning on each compressed subgraph to obtain the embedding vector of each node in the compressed subgraph; calculating the embedding vector, extracting the multi-scale features of the embedding vector, and perceiving the type of collaborative operation according to the multi-scale features. The present application forms a dynamic operation network by combining the collaborative agent and its trajectory data, the collaborative workstation and the operation scene, ensuring the consistency between the collaborative agent and the collaborative workstation, and retaining the temporal and spatial collaborative relationship between the collaborative agent and the collaborative workstation. By discretizing and compressing the dynamic operation network, the long-term spatial trajectory is compressed into multiple static short-term spatial trajectories, which is conducive to understanding the collaborative relationship between the collaborative agent and the collaborative workstation. Further, by random walks and learning, the embedding vector of the node is obtained, which can not only learn the features of each compressed subgraph, but also capture the temporal changes between different compressed subgraphs. Finally, we further learn the collaborative relationship between space and time, extract multi-scale features, and perform collaborative operation perception, avoiding the potential bias inherent in manually designed indicators. At the same time, this application has good portability in different operation scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1It is a flowchart of a collaborative operation perception method of the present application;

[0017] Figure 2 It is a schematic diagram of compression processing in a collaborative operation perception method of the present application;

[0018] Figure 3 It is a schematic diagram of the composition of a hierarchical self-attention mechanism in a collaborative task perception method of the present application;

[0019] Figure 4 This is a schematic diagram of user rating results of positive data in a collaborative work perception method of the present application;

[0020] Figure 5 This is a schematic diagram of user rating results of negative example data in a collaborative task perception method of the present application;

[0021] Figure 6 This is a schematic diagram of the accuracy of the cell side length and time step corresponding to the parameter sensitivity analysis in a collaborative operation perception method of the present application;

[0022] Figure 7 This is a schematic diagram of the results of the F1 score corresponding to the cell side length and the time step in the parameter sensitivity analysis in a collaborative operation perception method of the present application;

[0023] Figure 8 This is a schematic diagram of the composition of a collaborative operation sensing device of the present application;

[0024] Fig. 9 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present application;

[0025] Fig.10 It is a schematic block diagram of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0026] In order to facilitate the understanding of the present invention, the present invention is described in more detail below in conjunction with the accompanying drawings and specific embodiments. Preferred embodiments of the present invention are provided in the accompanying drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described in this specification. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

[0027] It should be noted that, unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used in this specification includes any and all combinations of one or more related listed items.

[0028] like Figure 1 As shown, the present invention provides a collaborative operation perception method, comprising the steps of:

[0029] S1: Obtain the trajectory data of the collaborative agent, and construct a dynamic operation network based on the collaborative workstations, collaborative agents and trajectory data; wherein the dynamic operation network includes multiple dynamic nodes and connecting edges.

[0030] S2: Discretize and compress the dynamic operation network in sequence to obtain multiple compressed subgraphs.

[0031] S3: Perform random walks and learning on each compressed subgraph to obtain the embedding vector of each node in the compressed subgraph.

[0032] S4: Calculate the embedded vector, extract the multi-scale features of the embedded vector, and perceive the type of collaborative work based on the multi-scale features.

[0033] This application forms a dynamic operation network by combining collaborative agents and their trajectory data, collaborative workstations and operation scenarios, ensuring the consistency between collaborative agents and collaborative workstations, and preserving the temporal and spatial collaborative relationship between collaborative agents and collaborative workstations. By discretizing and compressing the dynamic operation network, long-term spatial trajectories are compressed into multiple static short-term spatial trajectories, which is conducive to understanding the collaborative relationship between collaborative agents and collaborative workstations. Further, through random walks and learning, the embedding vectors of the nodes are obtained, which can not only learn the characteristics of each compressed subgraph, but also capture the temporal changes between different compressed subgraphs. Finally, the spatial and temporal collaborative relationship is further learned, multi-scale features are extracted, and collaborative operation perception is performed, avoiding the potential bias inherent in artificially designed indicators. At the same time, this application has good portability in different operation scenarios.

[0034] It should be noted that the method of the present application can be applied to the perception of collaborative operations of civil aircraft (such as material preparation, refueling, maintenance, baggage / cargo transfer, baggage / cargo caching, etc.). When applied to the civil field, the operation scene is an airport, the collaborative intelligent bodies are collaborative personnel and collaborative vehicles, and the collaborative workstations are resource positions, refueling positions, maintenance stations, parking positions, etc.). The embodiments of the present application mainly take the perception of collaborative operations of civil aircraft in the civil field as an example to specifically introduce the method of the present application.

[0035] Combine the following Figures 1 to 10 , the present application is further described in detail with specific embodiments.

[0036] Figure 1 The following is a flow chart of a collaborative operation perception method provided by an embodiment of the present application, which is described in detail as follows:

[0037] S1: Obtain the trajectory data of the collaborative agent, and construct a dynamic operation network based on the collaborative workstations, collaborative agents and trajectory data; wherein the dynamic operation network includes multiple dynamic nodes and connecting edges.

[0038] It should be noted that in this embodiment, the collaborative agent refers to the collaborative personnel and collaborative vehicles that continuously move and perform specific behaviors between different collaborative workstations in the work scene in order to complete the collaborative operation. In this embodiment, the collaborative workstation can be a resource position, a refueling position, a maintenance station or a parking position, etc. Correspondingly, the collaborative operation can be cargo transfer or refueling, etc., and the work scene is an airport.

[0039] In order to complete the collaborative work, the collaborative intelligent body will move in the work scene within a certain period of time, and the spatial position of the collaborative intelligent body will change over time. Therefore, after completing the collaborative work, a series of movement trajectory data will be generated.

[0040] First, the trajectory data of the collaborative intelligent body is collected through communication positioning technology, and a shared space-time cellular space is created based on the operation scenario and trajectory data.

[0041] Specifically, the shared spatiotemporal cellular space is a three-dimensional space constructed based on the work scene (considered as a two-dimensional plane) and the timeline of the trajectory data. The shared spatiotemporal cellular space includes multiple cells of the same size, each of which has a size of Δs = (Δx, Δx, Δt), where the first and second Δx represent the length and width of the cell on the two-dimensional plane of the work scene, respectively, and Δt represents the height of the cell on the time dimension of the trajectory data.

[0042] Furthermore, based on the shared spatiotemporal cellular space, the spatiotemporal (time and space) collaborative relationships between collaborative workstations and collaborative agents, and between different collaborative agents are calculated and obtained.

[0043] Specifically, if in the current time step, the collaborative workstation and the collaborative agent are in the same cell, or different collaborative agents are in the same cell, it proves that there is a spatial collaborative relationship between them at the current time; if they are not in the same cell, it proves that there is no spatial collaborative relationship between them at the current time.

[0044] In this embodiment, the spatiotemporal collaboration relationship can be expressed by the following formula:

[0045] E dyn (t)={E(Δt),E(2Δt),…,E(NΔt)}

[0046] Among them, E dyn(t) represents the spatiotemporal collaborative relationship, Δt represents one of the moments, NΔt represents the total duration of the trajectory data (i.e., the total time required to complete the collaborative task), and E(Δt) represents the spatial collaborative relationship at one of the moments.

[0047] Furthermore, the collaborative workstations and collaborative agents are taken as nodes, and the spatiotemporal collaborative relationships are taken as the connecting edges between the nodes to obtain a dynamic operation network. The network can be expressed by the following formula:

[0048] G dyn (t)= <V,E dyn (t)>

[0049] Among them, G dyn (t) represents the dynamic operation network, E dyn (t) represents the connecting edge (i.e., the spatiotemporal collaborative relationship), V represents the node (i.e., collaborative workstations and collaborative agents), and the number of V is the sum of the number of collaborative workstations and collaborative agents in the work scenario.

[0050] S2: Discretize and compress the dynamic operation network in sequence to obtain multiple compressed subgraphs.

[0051] It should be noted that the dynamic job network describes the entire collaborative job execution process, so the network can change dynamically over time.

[0052] Discrete processing is to divide the dynamic operation network into multiple static discrete subgraphs according to a certain time standard, and observe the connection relationship between nodes (i.e. spatial collaboration relationship) from different moments. For example, if the time standard is one second, the dynamic operation network is divided into discrete subgraphs of the first second, discrete subgraphs of the second second, and so on, discrete subgraphs of the Tth second. T is the total time required to complete the collaborative operation.

[0053] Then the spatial collaboration relationship at one moment can be expressed as:

[0054] E(kΔt)=(e ij k ) M×M

[0055] Among them, k represents the kth moment, e ij represents the connection edge between node i and node j, e ij k Represents the node v at the kth time i and node v j The connection edges between them, M represents the number of nodes, v i ,v j ∈V(i,j∈[1,M]), k∈[1,N].

[0056] Furthermore, the compression process includes: dividing the multiple discrete subgraphs into multiple compression groups according to the compression time step, and merging the multiple discrete subgraphs in each compression group into one compressed subgraph.

[0057] In this embodiment, combined with Figure 2 As shown in the figure, with Δt as a compression time step, there are multiple discrete sub-steps between [kΔt, (k+1)Δt) Figure 1 (For example, in kΔt, Discrete sub-time Figure 1 ) can be divided into a compression group, and the nodes (a, b, c, d) and the connection relationships ( Figure 2 The line segments between a, b, c, and d in the figure are merged to obtain a compressed sub Figure 2 Among them, a and d represent different collaborative personnel in the collaborative intelligent agent, b represents the aircraft, and c represents the collaborative workstation.

[0058] S3: Perform random walks and learning on each compressed subgraph to obtain the embedding vector of each node in the compressed subgraph.

[0059] In this embodiment, the random walk specifically includes:

[0060] For each compressed subgraph, a random walk of a target number γ is performed according to a first target length l; wherein each random walk obtains a random walk sequence.

[0061] For example, for one of the nodes v in one of the compressed subgraphs i , set the first target length l to 10 and the target number γ to 3, then you need to start from the current node v i Randomly walk to the next node until it has walked through 10 nodes, generating a random walk sequence accordingly. Then, perform the second random walk, also starting from the current node v i Initially, the random walk passes through 10 nodes until three random walks are completed, generating three corresponding random walk sequences.

[0062] Next, a sliding window with a second target length (s+1+s) is used to slide on each random walk sequence to obtain multiple node pairs.

[0063] For example, assuming that the second target length is (2+1+2=5, i.e., s=2), then taking one of the random walk sequences with 10 nodes as an example, a sliding window with a second target length of 5 is slid on the random walk sequence. If the sliding window slides to the third to seventh nodes, the five nodes in the sliding window can respectively form multiple node pairs (such as ).

[0064] Further studies include:

[0065] Through the skip-gram with negative-sample model (SGNS), multiple node pairs are trained and learned to obtain the training vector of each node pair. The above process can be expressed by the following formula:

[0066]

[0067] Among them, q represents the distribution from a univariate The number of negative samples extracted from , σ represents the Sigmoid function, node v t i、v t j and v t The training vectors of j' at time step t are and E represents the average value. t i、v t j is a positive node pair, node v t j、v t j' is a negative node pair. A positive node pair is a node pair in the same sliding window, and a negative node pair is a node pair not in the same sliding window.

[0068] Furthermore, the training vectors of each node pair in all compressed subgraphs are iteratively processed to obtain the embedding vectors of all nodes. The above process can be expressed by the following formula:

[0069]

[0070] Among them, Z k represents the embedding matrix at time kΔt (embedding vectors of all nodes), Represents the initialization embedding vector of the first compressed sub-graph, SGNS (k-1) represents the training vector of the previous compressed sub-graph, G'(kΔt) represents the current compressed sub-graph, and G'((k-1)Δt) represents the previous compressed sub-graph.

[0071] S4: Calculate the embedded vector, extract the multi-scale features of the embedded vector, and perceive the type of collaborative work based on the multi-scale features.

[0072] In this embodiment, combined with Figure 3 As shown in the figure, all embedding vectors are merged into a whole (EmbeddingVectors) and input into the hierarchical self-attention mechanism (Swin Transformer) for calculation.

[0073] Specifically, the merged embedding vector is divided into multiple data blocks through the Patch Partition module. After linear embedding multiple data blocks through the Linear Embedding module, the global features and local features of the data blocks are extracted through two Swin Transformer Block modules. Furthermore, after dimensionality reduction processing of global features and local features is performed through the Patch Merging module, multi-scale features are extracted through two Swin Transformer Block modules.

[0074] Finally, the extracted multi-scale features are input into the Fully Connected layer for collaborative perception. If there are three types of collaborative work (type 1, type 2, and type 3), the Fully Connected layer will output three numbers less than 1 (such as 0.2, 0.7, and 0.3) in sequence, and the second number 0.7 is the largest number among the three numbers, which means that the type of collaborative work is type 2.

[0075] In summary, reference Figures 4 to 7 As shown, in order to verify the performance of the collaborative work perception method of the present application, further verification is carried out through the following experiments.

[0076] First, in order to solve the problem of insufficient public availability of trajectory data for collaborative operations at airports, this application proposes a simulation-based method to generate a dataset. In collaboration with domain experts, a simulation scenario centered on cargo transfer was developed. A virtual airport operation environment was created using the Unreal Engine and the simulation algorithm was integrated. The main goal was to generate a reasonable trajectory dataset of collaborative agents in order to validate the method of this application.

[0077] Specific, combined Figures 4 to 5 As shown in the figure, we take cargo transfer (collaborative work) at the airport as an example. The destination of cargo transfer is the aircraft. Therefore, we also define the aircraft at the airport as a collaborative workstation.

[0078] Furthermore, the generated simulation datasets are mainly divided into two categories: one includes cargo transfer (positive data) and the other does not include cargo transfer (negative data). In a single data sample, there are 20 to 50 collaborative agents and 3 to 7 aircraft. For all collaborative agents, some are assigned to perform cargo transfer, and the rest are assigned to other collaborative tasks.

[0079] For the positive data, multiple scenarios are considered to ensure the diversity of trajectory data. The first scenario is that all collaborative agents start from the same standby area; the second scenario is that the collaborative agents start from different standby areas.

[0080] In addition, during the cargo transfer process, there are other types of collaborative agents that perform different collaborative tasks at the same time. There are three cases: the first case is a situation that does not intersect with the cargo transfer trajectory, the second case is a situation that intersects with the cargo transfer trajectory, and the third case is a situation that intersects with multiple cargo transfer trajectories.

[0081] Negative data lacks 1-2 features of cargo transfer and can be divided into six cases. The first case is: the collaborative agent does not take the cargo from the cargo buffer; the second case is: the collaborative agent does not go to the designated aircraft after taking the cargo; the third case is: a combination of the features of the first and second cases; the fourth case is: the collaborative agent does not stay in the cargo buffer; the fifth case is: the collaborative agent does not stay at the designated aircraft; the sixth case is: a combination of the fourth and fifth cases. Negative data also corresponds to some real situations at the airport: 1. When collaborative agents perform operations unrelated to cargo transfer, such as aircraft maintenance, refueling, etc. In these cases, they may not pass through the cargo buffer, that is, the first case, or they may just pass by briefly, that is, the fourth case. 2. When collaborative agents perform operations unrelated to aircraft, such as repairing fixed equipment at the airport. In these scenarios, they may not pass near the aircraft, that is, the third case, or they may just pass by briefly, that is, the sixth case. 3. Special circumstances lead to the interruption of cargo transportation, such as operation blockage, namely the second and fifth situations.

[0082] The following is the process of generating the dataset:

[0083] In the first stage, the total number of collaborative agents is initialized, of which 3 to 11 perform cargo transfers, and the rest perform other collaborative operations. The initial coordinates of these collaborative agents are randomly generated throughout the airport, and their destination coordinates are randomly assigned collaborative workstations. The locations of all aircraft parking stands at the airport are determined by experts. When the collaborative agents arrive at their assigned collaborative workstations, their operation duration usually ranges from a few minutes to tens of minutes near these locations. However, considering that the coordinates of the collaborative agents do not change much during this process, we optimize the data storage cost by shortening the operation duration in the simulation to 10 to 30 seconds. Including the travel time to the collaborative workstation, the simulation duration of a single data sample ranges from 300 seconds to 3000 seconds, captured at a rate of 25 frames per second.

[0084] In the second stage, motion simulation is performed based on the preliminary settings. When the collaborative agents start to move to the collaborative workstation, we use the Reciprocal Velocity Obstacles (RVO) algorithm. When they move to the collaborative workstation, we simulate the collaborative agents performing tasks.

[0085] The third stage is to visualize the simulation trajectory. In this process, the direction changes of the collaborative agents are smoothed to ensure that their movements look natural and smooth. In addition, the data where collisions occur is deleted to enhance the realism of the simulation.

[0086] Finally, data samples verified by experts as reliable are stored. The information recorded for each data sample includes trajectory data (frame ID, object ID, X-axis coordinate, Y-axis coordinate, X-axis speed (Vx), Y-axis speed (Vy)) and the categories and coordinates of the collaborative workstations of all collaborative agents. It should be noted that cargo transfer involves the trajectories of people and vehicles. Here, we do not distinguish between the trajectories of people and vehicles, and uniformly refer to them as collaborative agents.

[0087] Furthermore, the dataset was evaluated through user surveys: the users included 28 participants (15 males and 13 females) who were familiar with airport collaborative operations and engaged in related research. They evaluated the credibility of the dataset by observing the visualization results in the Unreal Engine. Metrics: Participants scored the data using a 7-point Likert scale, where 1 was very credible, 3 was relatively credible, and 7 was very uncredible. In our evaluation, a one-sample t-test was performed, assuming that the average credibility score was greater than 3, indicating that users believed the dataset was uncredible.

[0088] Figure 4 , Figure 5 The score results are shown. Figure 4 As shown in the figure, for four positive data describing cargo transshipment, the single sample t-test results are as follows:

[0089] Pos_Scenario1: t=-3.47, p<0.005;

[0090] Pos_Scenario2: t=-3.56, p<0.005;

[0091] Pos_Scenario3: t=-4.03, p<0.005;

[0092] Pos_Scenario4: t=-3.57, p<0.005.

[0093] like Figure 5 As shown in the figure, for three negative examples without cargo transshipment, the single sample t-test results are as follows:

[0094] Neg_Scenario1: t=-3.32, p<0.005;

[0095] Neg_Scenario2: t=-3.23, p<0.005;

[0096] Neg_Scenario3: t=-4.10, p<0.005.

[0097] These results suggest that participants generally believed that the constructed dataset was credible.

[0098] Further, combined with Figure 6 to Figure 7 As shown, the experimental results are presented from the following aspects: parameter sensitivity, functionality, performance of random walk and learning methods, performance of collaborative task awareness methods, ablation studies and transferability.

[0099] The perceptual results were evaluated using accuracy and F1 score. Accuracy measures the proportion of samples that were correctly predicted relative to the total number of samples. On the other hand, F1 score combines precision and recall for a more comprehensive evaluation. Precision measures the proportion of true positive samples among all samples predicted as positive, while recall measures the proportion of true positive samples among all actual positive samples. The F1 score represents the harmonic mean of precision and recall, providing a balanced evaluation of model performance.

[0100] The first is the functional analysis, and the evaluation results are shown in Table 1 below:

[0101] Table 1

[0102] Data types MEBN Ours The first positive case Yes Yes The second positive case No Yes The first negative case No Yes The second negative case Yes Yes The third negative case Yes Yes The fourth negative case No Yes The fifth negative case Yes Yes The sixth negative case Yes Yes

[0103] Table 1 analyzes the functionality of our approach and the existing collaborative work detection method MEBN. Among all positive data types, the performance of MEBN using the distance-based clustering algorithm is significantly insufficient when the collaborative agents start from different locations, especially in the second positive case. Regarding negative data, MEBN performs well in analyzing the relationship between the collaborative agents and the aircraft, and can accurately perceive the second negative case, the third negative case, the fifth negative case, and the sixth negative case. However, one drawback of MEBN is that it does not take into account the collaborative workstations on the airport. This leads to inaccurate perception of the first negative case and the fourth negative case, especially when the collaborative agents do not go to the collaborative workstations.

[0104] The second is parameter sensitivity analysis:

[0105] In order to study the impact of the parameters of the dynamic task network on the perception results, experiments were conducted by changing the values ​​of hyperparameters (i.e., the cell side length Δx and the time step Δt for constructing the dynamic task network). Figure 6 , Figure 7The effects of changing Δx from 400cm to 800cm and Δt from 4s to 10s on the final perception accuracy and F1 score are shown. A Δx that is too small cannot capture the complete topological relationship between collaborative agents, while a Δx that is too large will capture irrelevant topological structures. Similarly, a Δt that is too small will interfere with temporal information, while a Δt that is too large will result in the loss of some temporal details. Therefore, in subsequent experiments, we choose Δx to be 600cm and Δt to be 6s.

[0106] Next is the performance analysis of random walk and learning methods:

[0107] To verify the performance of the random walk and learning methods we used, we compared our random walk and learning methods with DeepWalk and Node2Vec on the same network. The three methods use the same set of parameters: γ = 10, l = 80, s + 1 + s = 10, d = 64. Given that DeepWalk and Node2Vec are static network algorithms, we learn the embedding vectors of the nodes for each compressed subgraph separately. Subsequently, the results obtained by the three methods are processed using the Swin Transformer configured with the same parameter settings. The final performance analysis is shown in Table 2 below:

[0108] Table 2

[0109] Method Accuracy F1 Score Ours 92.17 91.28 DeepWalk 63.86 64.98 Node2vec 66.87 64.36

[0110] Our method surpasses DeepWalk and Node2Vec in accuracy and F1 score. This advantage can be attributed to the fact that our method not only learns the features of each compressed subgraph, but also captures the temporal changes between different compressed subgraphs. In contrast, DeepWalk and Node2Vec can only capture the features of a single compressed subgraph because they are static algorithms.

[0111] The next step is the performance analysis of the collaborative operation perception method:

[0112] To verify the performance of our collaborative job perception method, we compare our method with Transformer. The inputs of the two neural networks are the same, and both come from the embedding vectors learned by applying our random walk and learning method to the same job network. The perception results of the two methods are presented in Table 3:

[0113] Table 3

[0114]

[0115]

[0116] As can be seen from Table 3, our method shows higher accuracy and F1 score than Transformer. Our method performs well in capturing features of different scales by leveraging its moving window mechanism.

[0117] Next is the ablation study:

[0118] Ablation studies do not perform random walks and learning methods (DynRLNet) on the dynamic job network. Considering the long-term nature of the trajectory, the trajectory data is directly input into the Swin Transformer at 6-second intervals. Table 4 shows the comparison results between directly inputting the trajectory and perceiving it from the perspective of the dynamic job network:

[0119] Table 4

[0120] Method Accuracy F1 Score Ours 92.17 91.28 Without DynRLNet 61.85 62.15

[0121] Obviously, our method achieves better results, which further proves the effectiveness of our method.

[0122] Further migration studies:

[0123] The transferability was analyzed to examine the ability on similar collaborative operations. Similar collaborative operations refer to two-stage operations similar to cargo transfers at airports. These operations usually involve going to the resource location to obtain the corresponding resources and then moving to the designated aircraft. Although the operation phases are generally similar, the number of collaborative agents and the duration of the operation execution are different. Two datasets were generated based on the above experiments: AgentNum, where the number of collaborative agents ranged from 1 to 3 times that of cargo transfers; TaskTime, where the duration of collaborative operations was 1 to 3 times longer than that of cargo transfers. The types of interference trajectories remained consistent. The model was trained on the first positive case (+ the 1st and 2nd interference cases) and the 1st to 6th negative cases (+ the 1st and 2nd interference cases), i.e., Dataset-1, and its transferability was evaluated on AgentNum and TaskTime. The transferability analysis of similar collaborative operations is shown in Table 7 below:

[0124] Table 7

[0125] Dataset Accuracy Dataset-1 92.17 AgentNum 80.56 TaskTime 80.77

[0126] As shown in Table 7, our method shows good transferability in different collaborative work scenarios.

[0127] Corresponding to the collaborative operation perception method in the above embodiment, Figure 8 A schematic diagram of the composition of a collaborative work perception device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0128] refer to Figure 8 , the collaborative operation sensing device comprises:

[0129] The acquisition construction unit 101 is used to obtain the trajectory data of the collaborative agent, and construct a dynamic operation network according to the collaborative workstation, the collaborative agent and the trajectory data; wherein the dynamic operation network includes a plurality of dynamic nodes and connecting edges.

[0130] The discrete compression unit 102 is used to perform discrete processing and compression processing on the dynamic operation network in sequence to obtain multiple compressed subgraphs.

[0131] The walking learning unit 103 is used to perform random walking and learning on each compressed subgraph to obtain an embedding vector of each node in the compressed subgraph.

[0132] The extraction and perception unit 104 is used to calculate the embedded vector, extract the multi-scale features of the embedded vector, and perceive the type of collaborative operation based on the multi-scale features.

[0133] It should be noted that since the above-mentioned collaborative work perception device is based on the same concept as the embodiment of the method of the present application, its specific functions, the process of each module realizing its respective functions and the technical effects brought about can be specifically referred to the description of the embodiment shown in the above-mentioned collaborative work perception method, and will not be repeated here.

[0134] Based on the same inventive concept, the present application also provides an electronic device, which includes a processor, a memory and a communication circuit, and the processor is connected to the memory and the communication circuit respectively; wherein the communication circuit is used for communication connection, the memory is used to store a computer program, and the processor is used to execute the computer program to implement the above-mentioned collaborative work perception method.

[0135] See also Fig. 9 The electronic device described in the embodiment of the present application may specifically include a processor 210 and a memory 220. The memory 220 is coupled to the processor 210.

[0136] The processor 210 is used to control the operation of the electronic device. The processor 210 may also be referred to as a CPU (Central Processing Unit). The processor 210 may be an integrated circuit chip having the ability to process signals. The processor 210 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor 210 may also be any conventional processor, etc.

[0137] The memory 220 is used to store computer programs, and may be a RAM, a ROM, or other types of storage terminals. Specifically, the memory 220 may include one or more computer-readable storage media, which may be non-transitory or transient. The memory 220 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage terminals, flash memory storage terminals. In some embodiments, the non-transitory computer-readable storage medium in the memory 220 is used to store at least one program code.

[0138] The processor 210 is used to execute the computer program stored in the memory 220 to implement the methods described in the various method embodiments of the present application.

[0139] In some embodiments, the electronic device may further include: a peripheral terminal interface 230 and at least one peripheral terminal. The processor 210, the memory 220 and the peripheral terminal interface 230 may be connected via a bus or a signal line. Each peripheral terminal may be connected to the peripheral terminal interface 230 via a bus, a signal line or a circuit board. Specifically, the peripheral terminal includes: at least one of a radio frequency circuit 240, a display screen 250, an audio circuit 260 and a power supply 270.

[0140] The peripheral terminal interface 230 may be used to connect at least one peripheral terminal related to I / O (Input / output) to the processor 210 and the memory 220. In some embodiments, the processor 210, the memory 220, and the peripheral terminal interface 230 are integrated on the same chip or circuit board; in some other implementations, any one or two of the processor 210, the memory 220, and the peripheral terminal interface 230 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0141] The radio frequency circuit 240 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 240 communicates with the communication network and other IoT devices through electromagnetic signals, and the radio frequency circuit 240 is the communication circuit of the electronic device. The radio frequency circuit 240 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 240 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, an operator identity module card, and the like. The radio frequency circuit 240 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 240 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0142] The display screen 250 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 250 is a touch display screen, the display screen 250 also has the ability to collect touch signals on the surface or above the surface of the display screen 250. The touch signal can be input to the processor 210 as a control signal for processing. At this time, the display screen 250 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 250 can be one, set on the front panel of the electronic device; in other embodiments, the display screen 250 can be at least two, respectively set on different surfaces of the electronic device or in a folding design; in other embodiments, the display screen 250 can be a flexible display screen, set on a curved surface or a folding surface of the electronic device. Even, the display screen 250 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 250 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).

[0143] The audio circuit 260 may include a microphone and a speaker. The microphone is used to collect sound waves from the operator and the environment, and convert the sound waves into electrical signals and input them into the processor 210 for processing, or input them into the radio frequency circuit 240 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged in different parts of the electronic device. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 210 or the radio frequency circuit 240 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 260 may also include a headphone jack.

[0144] The power supply 270 is used to power various components in the electronic device. The power supply 270 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 270 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged through a wired line, and a wireless rechargeable battery is a battery that is charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0145] For a detailed description of the functions and execution processes of each functional module or component in the electronic device embodiment of the present application, reference can be made to the description in the above-mentioned method embodiments of the present application, which will not be repeated here.

[0146] In the several embodiments provided in the present application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the various embodiments of the electronic devices described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0147] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0148] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0149] Based on the same inventive concept, the present application also provides a computer-readable storage medium storing a computer program, which can be executed by a processor to implement the above-mentioned collaborative work perception method.

[0150] See also Fig.10 , if the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium 300. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions / computer programs to enable an IoT device (which can be a personal computer, server, or network terminal, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks or optical disks, and electronic terminals such as computers, mobile phones, laptops, tablet computers, cameras, etc. having the above-mentioned storage media.

[0151] The description of the execution process of the program data in the computer-readable storage medium can refer to the description in the above-mentioned method embodiments of the present application, and will not be repeated here.

[0152] It can be seen that the present invention discloses a collaborative operation perception method, device, equipment and medium, which includes the steps of: obtaining the trajectory data of the collaborative intelligent body, constructing a dynamic operation network according to the collaborative workstation, the collaborative intelligent body and the trajectory data; discretizing and compressing the dynamic operation network in turn to obtain multiple compressed subgraphs; performing random walks and learning on each compressed subgraph to obtain the embedding vector of each node in the compressed subgraph; calculating the embedding vector, extracting the multi-scale features of the embedding vector, and perceiving the type of collaborative operation according to the multi-scale features. The present application forms a dynamic operation network by combining the collaborative intelligent body and its trajectory data, the collaborative workstation and the operation scene, ensuring the consistency between the collaborative intelligent body and the collaborative workstation, and retaining the time and space collaborative relationship between the collaborative intelligent body and the collaborative workstation. By discretizing and compressing the dynamic operation network, the long-term spatial trajectory is compressed into multiple static short-term spatial trajectories, which is conducive to understanding the collaborative relationship between the collaborative intelligent body and the collaborative workstation. Further, by random walks and learning, the embedding vector of the node is obtained, which can not only learn the characteristics of each compressed subgraph, but also capture the time changes between different compressed subgraphs. Finally, we further learn the collaborative relationship between space and time, extract multi-scale features, and perform collaborative operation perception, avoiding the potential bias inherent in manually designed indicators. At the same time, this application has good portability in different operation scenarios.

[0153] The above are only embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structural transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A collaborative operation perception method, characterized in that: Includes steps: Acquire trajectory data of the collaborative agent, and construct a dynamic operation network according to the collaborative workstation, the collaborative agent and the trajectory data; wherein the dynamic operation network includes a plurality of dynamic nodes and connecting edges; The dynamic operation network is sequentially subjected to discretization processing and compression processing to obtain a plurality of compressed subgraphs; Performing random walk and learning on each of the compressed subgraphs to obtain an embedding vector of each of the nodes in the compressed subgraph; The embedding vector is calculated, multi-scale features of the embedding vector are extracted, and the type of collaborative operation is perceived based on the multi-scale features.

2. The collaborative operation perception method according to claim 1, characterized in that: The construction of a dynamic operation network specifically includes: Creating a shared spatiotemporal cellular space according to the operation scenario and the trajectory data; According to the shared spatiotemporal cellular space, calculating and obtaining the spatiotemporal collaboration relationship between the collaborative workstation and the collaborative agent, and between different collaborative agents; The collaborative workstations and collaborative agents are used as the nodes, and the spatiotemporal collaborative relationships are used as connecting edges between the nodes to obtain the dynamic operation network.

3. The collaborative operation perception method according to claim 1, characterized in that: In the step of sequentially performing discretization processing and compression processing on the dynamic operation network, the discretization processing includes: The dynamic operation network is discretized in time to obtain a plurality of static discrete subgraphs.

4. The collaborative operation perception method according to claim 3, characterized in that: In the step of sequentially performing discretization processing and compression processing on the dynamic operation network, the compression processing includes: The plurality of discrete subgraphs are divided into a plurality of compression groups according to the compression time step, and the plurality of discrete subgraphs in each compression group are merged into one compression subgraph.

5. The collaborative operation perception method according to claim 1, characterized in that: In the step of performing random walk and learning on each of the compressed subgraphs, the random walk specifically includes: For each of the compressed subgraphs, a target number of random walks are performed according to a first target length; wherein each random walk obtains a random walk sequence; A sliding window of a second target length is used to slide on each of the random walk sequences to obtain a plurality of node pairs.

6. The collaborative operation perception method according to claim 5, characterized in that: In the step of performing random walk and learning on each of the compressed subgraphs, the learning specifically includes: Using a skip-word negative sampling model, training and learning are performed on the plurality of node pairs to obtain a training vector for each of the node pairs; The training vectors of each of the node pairs in all the compressed subgraphs are iteratively processed to obtain the embedding vectors of all the nodes.

7. The collaborative operation perception method according to claim 1, characterized in that: The calculating the embedded vector to extract the multi-scale features of the embedded vector specifically includes: Merging all the embedding vectors into a whole and dividing them into multiple data blocks; After linearly embedding the plurality of data blocks, extracting global features and local features of the data blocks; After performing dimensionality reduction processing on the global features and the local features, the multi-scale features are extracted.

8. A collaborative operation sensing device, characterized in that: include: An acquisition construction unit is used to obtain trajectory data of the collaborative agent, and construct a dynamic operation network according to the collaborative workstation, the collaborative agent and the trajectory data; wherein the dynamic operation network includes a plurality of dynamic nodes and connecting edges; A discrete compression unit, used for sequentially performing discrete processing and compression processing on the dynamic operation network to obtain a plurality of compressed subgraphs; A walking learning unit, used for performing random walking and learning on each of the compressed subgraphs to obtain an embedding vector of each of the nodes in the compressed subgraphs; The extraction perception unit is used to calculate the embedding vector, extract the multi-scale features of the embedding vector, and perceive the type of collaborative operation based on the multi-scale features.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images