Unmanned station intelligent control method and program product based on multi-modal semantic modeling

Abstract

The application provides an unmanned station intelligent control method and program product based on multi-modal semantic modeling. In view of the complex operating environment often existing in an unmanned station, the current operating state of the unmanned station is uniformly perceived through real-time multi-modal data, and a unified semantic space is constructed through cross-modal semantic modeling to realize associated understanding and overall state cognition among multi-source heterogeneous data. On this basis, an event object oriented to station control is constructed, and in combination with a knowledge graph and a station operating context, dynamic reasoning and constraint are performed on a complex operating scenario to generate a target action execution chain adapted to the current operating state, and operation scheduling and cooperative control of an execution unit under the target action execution chain are implemented. At the same time, the target action execution chain is dynamically re-updated according to an execution error in the cooperative execution of the execution unit to realize closed-loop optimization, and the cooperative control efficiency, operating stability and self-adaptive ability of the unmanned station in a complex operating environment are improved.

Description

Technical Field

[0001] This application relates to the field of intelligent rail transit technology, specifically to an intelligent control method and program product for unmanned stations based on multimodal semantic modeling. Background Technology

[0002] With the development of smart rail transit, intelligent operation and maintenance, and unmanned operation technologies, some subway stations, logistics parks, and closed park scenarios have begun to explore unmanned or minimally staffed operation modes. The application of unmanned operation technology in subway stations is expected to improve operational efficiency and safety by assisting in anomaly handling, equipment linkage, and operation scheduling through real-time perception of equipment status, environmental status, and operational tasks.

[0003] However, in the existing implementation, although multimodal data from different devices can be obtained, such as data from turnstiles, surveillance cameras, various sensors, business systems, etc., the lack of effective correlation between the various modal data and the difficulty in forming an overall state understanding of the complex operating scenarios that often exist in unmanned stations can easily lead to problems such as inaccurate event recognition, delayed abnormal response, and low efficiency.

[0004] On the other hand, most of the control methods for abnormal responses rely on fixed rules or preset procedures. When the operating status of the subway station changes or the control effect is not good, it is difficult to dynamically adjust according to the real-time operating status, resulting in insufficient adaptive capability and affecting the overall operational stability of the unmanned station.

[0005] Therefore, how to improve the overall state awareness and adaptive operation capabilities of unmanned stations in complex operating scenarios has become a pressing technical problem that needs to be solved for the implementation of unmanned stations. Summary of the Invention

[0006] One objective of this application is to improve the overall state awareness and adaptive operation capabilities of unmanned stations in complex operating scenarios.

[0007] According to one aspect of the embodiments of this application, an intelligent control method for unmanned stations based on multimodal semantic modeling is disclosed, the method comprising:

[0008] Collect real-time multimodal data from unmanned stations, wherein the real-time multimodal data includes at least video data, sensor data, and text data; By mapping real-time multimodal data to a unified semantic space, a unified semantic representation is obtained. The unified semantic space is constructed through multi-level alignment training across modalities, which includes local alignment, bidirectional alignment, and global alignment. Based on the unified semantic representation, an event object is constructed for the unmanned station that currently triggers station control events. The event object includes at least status information, causal relationship fields, and impact information. For the event object, a set of candidate actions is generated by matching the knowledge graph and constraining the station operation context information. Based on the correlation and temporal dependency between the candidate actions, the target action execution chain is selected from the set of candidate actions. The execution unit implements the coordinated execution of the unmanned station through the target action execution chain, and updates the target action execution chain again according to the execution error between the target execution result and the actual execution result, so as to optimize the coordinated execution of the unmanned station in a closed loop.

[0009] According to one aspect of the embodiments of this application, a computer device is disclosed, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method as described in any of the preceding claims.

[0010] According to one aspect of the embodiments of this application, a computer program product is disclosed, including a computer program that, when executed by a processor, implements the steps of the method as described in any of the preceding claims.

[0011] This application's embodiments are geared towards the practical implementation of unmanned stations, constructing an intelligent control architecture that integrates perception, understanding, decision-making, execution, and feedback to address the problems of state fragmentation, difficulties in collaborative scheduling, and insufficient control adaptive capabilities in existing unmanned stations operating in complex environments.

[0012] This application applies to the operation of unmanned stations, addressing the complex operating environments that unmanned stations frequently encounter. It utilizes real-time multimodal data such as video, sensor, and text data to achieve unified perception of the station's current operating status. Furthermore, it constructs a unified semantic space through cross-modal semantic modeling, enabling the understanding of relationships between multi-source heterogeneous data and overall status cognition. Based on this, it constructs event objects for station control and, combined with knowledge graphs and station operating context, dynamically infers and constrains complex operating scenarios to generate target action execution chains adapted to the current operating status. This allows for the scheduling and collaborative control of execution units within the target action execution chain. Simultaneously, during the collaborative execution of execution units, the target action execution chain is dynamically updated based on the execution error between the target execution result and the actual execution result, achieving closed-loop optimization of the control strategy and improving the collaborative control efficiency, operational stability, and adaptability of unmanned stations in complex operating environments.

[0013] In summary, compared with traditional control systems based on fixed rules, the embodiments of this application can ensure the implementation of unmanned stations, and due to their unified state cognition, dynamic coordination capabilities, and feedback self-optimization capabilities during execution, they can effectively improve the intelligent operation level of unmanned stations.

[0014] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0015] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application. Attached Figure Description

[0016] The above and other objectives, features and advantages of this application will become more apparent from a detailed description of exemplary embodiments thereof with reference to the accompanying drawings.

[0017] Figure 1 This is a flowchart illustrating an intelligent control method for unmanned stations based on multimodal semantic modeling, according to an exemplary embodiment.

[0018] Figure 2 It is based on Figure 1 The corresponding embodiment shows a flowchart describing the steps of constructing an event object that triggers a station control event at an unmanned station based on a unified semantic representation.

[0019] Figure 3 This is a schematic diagram of the operation of the robot dogs deployed inside the unmanned station.

[0020] Figure 4 It is based on Figure 1 The flowchart in the corresponding embodiment describes the execution process of the execution unit in the unmanned station through the target action execution chain in step S150.

[0021] Figure 5 It is based on Figure 1 The flowchart in step S150 of the corresponding embodiment describes the method for updating the target action execution chain based on the execution error between the target execution result and the actual execution result to optimize the collaborative execution of the unmanned station in a closed loop.

[0022] Figure 6 It is based on Figure 1 The flowchart in the corresponding embodiment describes the execution process of generating a set of candidate actions for an event object through knowledge graph matching and constraints of station operation context information in step S140.

[0023] Figure 7 It is based on Figure 1The corresponding embodiment shows a flowchart describing the execution process of selecting the output target action execution chain from the candidate action set based on the correlation and temporal dependency between candidate events in step S141.

[0024] Figure 8 This is a flowchart illustrating an intelligent control method for unmanned stations based on multimodal semantic modeling, according to another exemplary embodiment.

[0025] Figure 9 It is based on Figure 8 The corresponding embodiment shows a flowchart describing the steps of constructing an alignment optimization objective based on a cross-modal similarity matrix, and jointly training a cross-modal coding model through local feature alignment, bidirectional matching constraints, and global semantic consistency constraints to obtain a cross-modal semantically aligned coding model.

[0026] Figure 10 This is a flowchart illustrating an intelligent control method for unmanned stations based on multimodal semantic modeling, according to another embodiment. Detailed Implementation

[0027] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided to make the description of this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art. The drawings are merely illustrative of this application and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0028] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more exemplary embodiments. Numerous specific details are provided in the following description to give a full understanding of exemplary embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced with one or more of the specific details omitted, or other methods, components, steps, etc., can be employed. In other instances, well-known structures, methods, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0029] Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0030] In the advancement of intelligent rail transit, unmanned or minimally staffed operation modes of subway stations have become an important component of the intelligent rail transit system. This application aims to provide a corresponding multimodal semantic model deployment and intelligent collaborative control architecture for unmanned or minimally staffed operation modes of subway stations. Through real-time perception of station operating status, unified semantic understanding, dynamic event reasoning, and collaborative control, intelligent operation control in complex operating scenarios can be achieved.

[0031] See Figure 1 , Figure 1 This is a flowchart illustrating an intelligent control method for unmanned stations based on multimodal semantic modeling, according to an exemplary embodiment.

[0032] The intelligent control method for unmanned stations based on multimodal semantic modeling provided in this application includes: Step S110: Collect real-time multimodal data from the unmanned station. The real-time multimodal data includes at least video data, sensor data, and text data. Step S120: By mapping real-time multimodal data to a unified semantic space, a unified semantic representation is obtained. The unified semantic space is constructed through multi-level alignment training across modalities. Multi-level alignment across modalities includes local alignment, bidirectional alignment, and global alignment. Step S130: Construct an event object for the unmanned station that currently triggers station control events based on a unified semantic representation. The event object includes at least status information, causal relationship fields, and impact information. Step S140: For the event object, a set of candidate actions is generated by matching the knowledge graph and constraining the station operation context information. Based on the correlation and temporal dependency between the candidate actions, the execution chain of the output target action is selected from the set of candidate actions. Step S150: Implement the collaborative execution of the execution unit in the unmanned station through the target action execution chain, and update the target action execution chain again according to the execution error between the target execution result and the actual execution result, so as to optimize the collaborative execution of the unmanned station in a closed loop.

[0033] These steps are explained in detail below.

[0034] This application's embodiments are applied to various subway stations of intelligent rail transit systems to achieve unmanned or minimally staffed operation modes in these stations. Utilizing the video surveillance equipment, environmental sensing equipment, equipment status sensors, and business systems already deployed at the subway stations, these serve as multiple data acquisition terminals to achieve real-time multimodal data acquisition in unmanned stations.

[0035] In step S110, the collected real-time multimodal data includes, but is not limited to, video data, sensor data, and text data. Video data is the output of the subway station video surveillance equipment. Sensor data is collected through various sensors, such as infrared passenger flow counters, turnstile systems, carbon dioxide sensors, escalator sensors, elevator status sensors, smoke detectors, and access control sensors, and can be used to characterize passenger flow, the environment, and even equipment and security status. Business systems, such as ticketing systems, dispatching systems, and log systems, collect text data.

[0036] Step S110 involves real-time access to multimodal data and time synchronization processing to generate real-time multimodal data corresponding to the current operating status of the unmanned station.

[0037] For example, in actual operation, video data can include videos of personnel activities, equipment operation, and channel monitoring; sensor data can include scheduling logs, abnormal alarm information, and operation and maintenance text records. Thus, unified access to real-time multimodal data helps to improve the overall perception of the complex operating status of unmanned stations.

[0038] In step S120, by mapping real-time multimodal data to a unified semantic space, a unified semantic representation of the real-time multimodal data is obtained under the action of the trained and deployed cross-modal coding model.

[0039] It should be understood that a cross-modal coding model is pre-built through cross-modal multi-level alignment training, which includes local alignment, bidirectional alignment and global alignment, to establish cross-modal semantic association capabilities between different modal data.

[0040] During the operation phase, as real-time modal data is acquired, step S120 performs feature encoding on each modal data to obtain corresponding modal features. Among these, video data can be encoded to obtain video semantics. Subsequently, under the action of the cross-modal coding model, a unified semantic mapping is performed on the different modal features, projecting each modal feature into a unified semantic space to obtain a unified semantic representation corresponding to the current station operation status.

[0041] Local alignment is used to establish correspondences between local semantic units of different modalities. For example, local semantic association is performed between local region features in a video and alarm semantic segments in text or local state features in a sensor, so that local features corresponding to the same operating state in different modalities are brought closer to each other in a unified semantic space.

[0042] Bidirectional alignment is used to enhance the bidirectional semantic consistency between different modalities, enabling video features to be associated with corresponding text features and sensor features, while text features and sensor features can also be associated with corresponding video features, thereby reducing the semantic bias problem caused by one-way mapping.

[0043] Global alignment is used to establish cross-modal semantic associations at the overall operational status level, enabling different modal data to not only form correspondences at the local state level, but also to form a unified semantic expression at the overall operational level. For example, equipment status, environmental status, and operational logs can be jointly mapped to a global semantic representation corresponding to the current station operational status.

[0044] By using a cross-modal, multi-layer alignment approach, we can enhance our overall understanding of complex operational scenarios and improve the accuracy of subsequent station control event identification and collaborative control decisions.

[0045] To further explain the cross-modal alignment training, the unified semantic space is obtained through multi-layer cross-modal alignment training. Local alignment can establish the correspondence between local semantic units of different modalities, bidirectional alignment can enhance the bidirectional semantic consistency between different modalities, and global alignment can establish cross-modal semantic associations at the overall operational state level. This enables unified semantic understanding between data from different modalities, allowing for an overall state cognition of complex operational scenarios at unmanned stations, thereby reducing the state fragmentation problem caused by single-modal analysis.

[0046] The unified semantic representation corresponds to the station control event currently triggered at the unmanned station. In other words, the station control event currently triggered at the unmanned station can be identified through state recognition and state correlation analysis based on the obtained unified semantic representation. Therefore, after obtaining the unified semantic representation by mapping the real-time multimodal data corresponding to the current operating state of the unmanned station, a structured expression of the station control event currently triggered at the unmanned station can be constructed based on the unified semantic representation, that is, the construction of the corresponding event object is performed.

[0047] In summary, in one embodiment, the execution process of step S120 includes: in response to the input of real-time multimodal data during the operation phase, performing feature encoding on each modal data in the real-time multimodal data to obtain the corresponding modal features; Based on the unified semantic space constructed through cross-modal multi-layer alignment training, modal features are mapped to generate unified semantic representations with cross-modal semantic consistency. The mapping steps of modal features to the unified semantic space include using the cross-modal coding model optimized through local alignment, bidirectional alignment and global alignment during the training phase to project different modal features to the unified semantic space, so that semantically consistent modal features will have similar representations in the unified semantic space.

[0048] In step S130, an event object that triggers the station control event at the unmanned station is constructed based on a unified semantic representation, thereby realizing the identification and structured expression of the station control event corresponding to the current operating state.

[0049] Station control events are used to characterize operational events that require coordinated control processing during the operation of an unmanned station. Examples of station control events include equipment malfunction events, congestion events, environmental risk events, scheduling conflict events, and safety risk events. Correspondingly, event objects are used to structure and represent station control events, enabling subsequent execution processes to perform reasoning, decision-making, and coordinated control based on the structured event information.

[0050] The event object includes at least state information, causal relationship fields, and impact information. State information includes the state label of the station control event and its quantified parameter values. Causal relationship describes the correlation between different states. For example, the causal relationship field includes at least the cause state, the result state, and the correlation strength. In the causal relationship field of an event object, the cause state is gate delay, and the result state is passenger congestion; the correlation strength between the two states is 0.87. This causal relationship field indicates the two related states and the degree of correlation between them. As another example, in the causal relationship field of an event object, the cause state is train delay, and the result state is platform congestion; the correlation strength is 0.9. In other words, there is a necessary correlation between the two states: train delay and platform congestion; that is, train delay inevitably leads to platform congestion, and the two have a high correlation. Impact information describes the scope and degree of impact of the current station control event on equipment units, operating areas, and scheduling tasks.

[0051] In step S130, event state identification is performed based on unified semantic representation to extract state features corresponding to the current operating state of the unmanned station. Adapted to the dimension of the modality, these state features include abnormal state features and operating state features. For example, based on unified semantic representation, abnormal state features and operating state features existing in the current station operation are extracted to roughly determine whether there are operating events requiring coordinated control processing at the current station, that is, to achieve the identification of initial events and obtain the initial event category.

[0052] In an exemplary embodiment, the unified semantic representation, as previously described, is a semantically consistent representation of various modal data, such as video data, sensor data, and text data. Therefore, the unified semantic representation contains the event states corresponding to each modality. Thus, in the performed event state recognition, after extracting state features from the event states of each dimension contained in the unified semantic representation, association recognition is performed. The semantic aggregation result in the unified semantic representation obtained from the association recognition is the initial event category existing in the current station operation. As previously described, the state features include abnormal state features and operational state features. In other words, the identified state features correspond to a specific event state represented in the unified semantic representation. This specific event state includes abnormal states, operational states, and / or environmental change states, etc., corresponding to at least one initially identified event category, which corresponds to either an abnormal state feature or an operational state feature.

[0053] For example, when the state features extracted from the unified semantic representation of multimodal data, such as the regional personnel aggregation features corresponding to video data, the equipment temperature rise features corresponding to sensor data, and the equipment anomaly alarm features corresponding to text data, have semantic associations in the unified semantic space, it is possible to identify the initial event categories related to equipment anomalies in the current station operation.

[0054] State characteristics, such as abnormal state characteristics and running state characteristics, are used to describe the specific state manifestations existing in the current running state. Therefore, by obtaining state characteristics, at least one initial event category can be further determined.

[0055] To further clarify, the initial event category is used to represent the basic event type that may correspond to the current operating state, and it is also the initial classification result of the station control events that the unmanned station will trigger. It should be understood that for the operation of an unmanned station, it is necessary to accurately perceive the station control events that will be triggered in the current operating state in order to provide a basis for subsequent reasoning and decision-making.

[0056] It should be further explained that through event state identification, at least one initial event category will be determined based on the obtained state characteristics. This initial event category is used to represent the basic event type that may correspond to the current operating state, and is the initial classification result of station control events, providing a basis for event categories for subsequent event objects. The complex operating scenarios in unmanned stations exhibit multi-state coupling phenomena; therefore, multiple initial event types may correspond simultaneously.

[0057] For example, the state characteristics describing the specific state manifestations existing in the current operating state may include equipment high temperature characteristics, equipment vibration abnormal characteristics, regional personnel aggregation characteristics, regional traffic abnormal characteristics, smoke abnormal characteristics, equipment communication abnormal characteristics, or scheduling task blocking characteristics, etc. Therefore, based on multiple state characteristics, at least one initial event category will be determined.

[0058] Furthermore, the state features, such as the obtained abnormal state features and running state features, have a semantic matching relationship with the preset event categories. The extracted state features can be mapped to event categories to determine at least one corresponding initial event category.

[0059] See Figure 2 , Figure 2 It is based on Figure 1 The corresponding embodiment shows a flowchart describing the steps of constructing an event object that triggers a station control event at an unmanned station based on a unified semantic representation.

[0060] The step S130 of this application embodiment, which constructs an event object for the currently triggered station control event of an unmanned station based on a unified semantic representation, includes: Step S131: Extract the current abnormal state features and operating state features of the unmanned station by performing event state recognition with unified semantic representation, and obtain at least one initial event category; Step S132: Generate state labels for multi-label recognition of the initial event category and quantization parameters for the state labels for multiple state attributes corresponding to at least one initial event category; Step S133: Using the status label and its quantization parameters, construct the status information, causal relationship field, and impact information of the station control event currently triggered by the unmanned station to form a structured event object.

[0061] These steps are explained in detail below.

[0062] The execution process of event state identification and extraction of state features in step S131 is as described above. Furthermore, at least one initial event category is obtained by semantic mapping of the obtained abnormal state features and operating state features of the unmanned station.

[0063] Based on this, multi-label recognition will be performed on the state attributes corresponding to at least one initial event category to obtain the corresponding state labels and their corresponding quantization parameter values. The state attributes describe the possible state dimensions that may be included under the corresponding initial event category. For example, for a device malfunction event, the corresponding state attributes include device temperature status, device load status, device vibration status, and device communication status; for a regional congestion event, the corresponding state attributes may include regional population status, regional traffic speed status, and regional dwell time status.

[0064] Based on the corresponding status attributes, multi-label recognition is performed on the current operating status to generate status labels. The status labels will identify the specific status content of the current station operation. For example, under equipment abnormality events, status labels such as "equipment high temperature", "equipment overload", "equipment vibration status" and "equipment communication status" are identified; under regional congestion events, status labels such as "people gathering", "passage speed decrease" and "regional congestion" are identified.

[0065] Correspondingly, the status labels are further quantified to generate corresponding quantized parameter values. These quantized parameter values ​​describe the degree or intensity of the status for each label. For example, the quantized parameter value for the "Equipment High Temperature" status label is the real-time temperature of the equipment; the quantized parameter value for the "Crowd Gathering" status label is the area's population density; and the quantized parameter value for the "Decreased Traffic Speed" status label is the area's average traffic speed.

[0066] After obtaining the status label and its quantization parameter values, an event object for structured event representation is constructed. The event object includes at least status information, causal relationship fields, and impact information. The event object is the data representation of the station control event currently triggered by the unmanned station.

[0067] The construction of event objects, i.e. the execution process of event modeling, firstly involves determining the state associations and event propagation relationships based on state labels. Then, causal association fields are constructed using these state associations and event propagation relationships. State associations characterize the state labels associated with a state, while event propagation relationships describe the event propagation path. Next, impact information is generated based on the causal association fields, the quantized parameter values ​​of the associated state labels, and the station operation context. Finally, using the state labels and their quantized parameter values ​​as state information, a structured event representation is constructed based on the state information, the causal association fields, and their semantically mapped event types and impact information to obtain the event object.

[0068] For status labels, further determine the status associations and event propagation relationships between them. Status associations describe the correlation between different status labels. For example, there is an equipment operation association between the "Equipment High Temperature" status label and the "Equipment Overload" status label; there is a regional operation association between the "Personnel Gathering" status label and the "Traffic Speed ​​Decreased" status label.

[0069] Event propagation relationships describe the propagation paths between devices, regions, and / or scheduling tasks. For example, overheating of a device may cause it to malfunction, which in turn triggers regional rate limiting, and regional rate limiting may further delay scheduling tasks. This process corresponds to an event propagation relationship.

[0070] For example, based on the temporal association, device association, regional association, and operational logic relationship between status tags, the existing status association and event propagation relationship are determined for all status tags.

[0071] It should be understood that based on state association, we can determine the interrelated state labels; and based on event propagation, we can obtain the event propagation path of the state label. Thus, based on event propagation, for a state label, we can determine the causal state to which it belongs, that is, the state label as the cause, and the result state mapped when it is the causal state, that is, other state labels.

[0072] For example, if a device malfunction label appears and a crowd gathering label appears in the corresponding area within a preset time window, it can be determined that there is an event propagation relationship between the device malfunction label and the area congestion label.

[0073] After determining the state relationships and event propagation relationships, the corresponding causal relationship fields are further constructed. These causal relationship fields are used to provide a structured description of the causal influence paths between different states within an event object, i.e., the event propagation paths. For example, in the event propagation chain of "equipment high temperature - equipment abnormal shutdown - regional flow restriction - scheduling task delay," corresponding causal relationship fields can be constructed to describe the causal propagation paths between events.

[0074] The obtained status labels and their quantified parameter values ​​are used as status information. Corresponding impact information is then generated based on causal correlation fields and station operational context information. Station operational context information describes the overall operational environment of the current station, and its sources include, but are not limited to, current scheduling tasks, equipment operating load, regional resource occupancy, train arrival and departure status, passenger flow status, and execution unit operational status. By introducing station operational context information, the constructed impact information can demonstrate different degrees of influence on the same event under different operational environments.

[0075] For example, the impact of the same level of population gathering in a region on station operations differs significantly between peak and off-peak hours; similarly, the same equipment malfunction may pose a higher operational risk under high load conditions. Therefore, by combining the current station operational context information, the scope and degree of impact of the current event can be dynamically generated to obtain impact information.

[0076] Thus, the obtained impact information, status information, causal relationship fields, and event types semantically mapped by status information form event objects, which provide a structured expression of the station control events to be triggered at the unmanned station.

[0077] In this way, the embodiments of this application can realize the construction of structured events in complex operating scenarios based on unified semantic representation. It can not only identify the current abnormal operating state, but also analyze the correlation between different states and the event propagation relationship, thereby improving the dynamic analysis and collaborative control capabilities of unmanned stations in complex operating scenarios, and providing a structured event basis for the generation of subsequent target action execution chains.

[0078] After completing the event modeling, step S140 generates a set of candidate actions for the event objects through knowledge graph matching and constraints from station operation context information. As mentioned earlier, station operation context information includes equipment operating status, regional resource occupancy, current scheduling tasks, and execution unit execution status. Therefore, under the influence of station operation context information, the resulting set of candidate actions can adapt to complex operating environments.

[0079] After obtaining candidate actions by matching the knowledge graph, the station operation context information is used as a constraint to eliminate candidate actions that are not applicable to the station operation context information, thus obtaining a set of candidate actions.

[0080] For the obtained set of candidate actions, based on the correlation and temporal dependency between the candidate actions, an output target action execution chain is selected from the set of candidate actions. The target action execution chain contains interconnected target actions. That is, for the obtained set of candidate actions, several candidate actions to be executed are selected based on the correlation and temporal dependency between the candidate actions. The selected candidate actions are the target actions executed in response to station control events. Then, based on the correlation and temporal dependency between the target actions, a target action execution chain adapted to the current operating state is generated for the target actions.

[0081] This approach enables dynamic collaboration in complex scenarios at unmanned stations, thereby improving the linkage capability and scheduling efficiency of the unmanned station control process.

[0082] For the target action execution chain obtained in step S140, step S150 implements the collaborative execution of the execution unit in the unmanned station, and updates the target action execution chain again based on the execution error between the target execution result and the actual execution result, so as to optimize the collaborative execution of the unmanned station in a closed loop.

[0083] First, it should be clarified that the execution units referred to are at least various embodied robots deployed in unmanned stations. For example, execution units include embodied robots such as inspection robots and robotic dogs, access control equipment, broadcasting equipment, and dispatching terminals, such as... Figure 3 The robot dog shown.

[0084] During execution, the execution feedback data of the execution unit is acquired in real time to obtain the actual execution result. Then, based on the execution error between the target execution result and the actual execution result, the target action execution chain is dynamically updated.

[0085] As target actions distributed along the target action execution chain are assigned to corresponding execution units, the collaborative execution, for example, may result in incomplete execution of some target actions, excessively high execution latency, or resource conflicts. In such cases, the action sequence, execution path, or execution unit allocation strategy of the target action execution chain may be readjusted.

[0086] Furthermore, it should be clarified that the target action execution chain is used to describe a set of related and time-dependent actions generated in response to the current station control event. For example, the target actions in a target action execution chain include area flow restriction actions, equipment power-off actions, area broadcast actions, inspection actions, task scheduling actions, and backup equipment switching actions.

[0087] Correspondingly, for the target action to be performed, the execution units are matched. The matched execution units include access control equipment, broadcasting equipment, inspection robots, dispatch terminals, environmental control equipment, and backup control terminals.

[0088] Furthermore, for the matching and coordinated execution of multiple execution units, see [reference needed]. Figure 4 , Figure 4 It is based on Figure 1 The flowchart in the corresponding embodiment describes the execution process of the execution unit in the unmanned station through the target action execution chain in step S150.

[0089] The execution process provided by step SE150 in this embodiment of the application, which implements the collaborative execution of the execution unit in the unmanned station through the target action execution chain, includes: Step S151a: Convert the target action execution chain into an embodied task chain to obtain a multi-execution unit collaborative task; Step S152a: For the target actions contained in the multi-execution unit collaborative task, perform execution unit matching to determine the execution units and the execution scheduling sequence of the execution units; Step S153a: For the target actions contained in the multi-execution unit collaborative task, perform execution unit matching to determine the execution units and the execution scheduling sequence of the execution units; Step S154a: Control the matched execution units to execute the target action serially or in parallel according to the execution scheduling sequence.

[0090] The obtained target action execution chain is transformed into an embodied task chain to obtain a multi-execution unit collaborative task. The transformation from the target action execution chain to the embodied task chain is the process of converting the target action in the target action execution chain into an entity task that can be executed by a specific execution unit, thereby obtaining a multi-execution unit collaborative task.

[0091] For example, the "area flow restriction - area broadcasting - inspection confirmation" chain in some target action execution chains can be further converted into the area flow restriction task being performed by the access control equipment; the area broadcasting task being performed by the broadcasting equipment; and the on-site inspection task being performed by the inspection robot.

[0092] In this way, the abstract target action execution chain is transformed into an executable task oriented towards a specific execution terminal. After obtaining the multi-execution unit collaborative task, execution units are matched for the target actions contained in the multi-execution unit collaborative task to determine the corresponding execution units and the execution scheduling sequence of the execution units.

[0093] Specifically, execution unit matching is the process of selecting the corresponding execution unit for a target action based on the current execution unit's operating status, device load, regional location, communication status, and task capabilities.

[0094] For example, when multiple inspection robot dogs deployed at an unmanned station are available at the same time, the robot dog that is closest to the target area to be inspected and has a low current load will be selected first to perform the corresponding inspection task during the matching process.

[0095] The execution scheduling sequence describes the order of action execution and the coordination relationship between the matched and determined execution units. Therefore, the execution scheduling sequence is generated based on the dependencies, action priorities, and resource consumption relationships between target actions. For example, in the response to a station control event of equipment malfunction, under the action of the generated execution scheduling sequence, the target action of "area flow restriction" is executed first, followed by the target action of "equipment power failure," to avoid directly cutting off power when personnel are still in the danger zone.

[0096] To further clarify, for target actions that do not have strong temporal dependencies, the corresponding execution units will be controlled to execute the target actions in parallel under the execution scheduling sequence. For example, the "area broadcast" action and the "backup inspection equipment scheduling" action can be executed simultaneously by different execution units to reduce overall response latency.

[0097] The above methods enable collaborative control and coordinated execution among multiple execution units in an unmanned station.

[0098] With the coordinated execution of the target action execution chain, see [reference] Figure 5 , Figure 5 It is based on Figure 1 The flowchart in step S150 of the corresponding embodiment describes the method for updating the target action execution chain based on the execution error between the target execution result and the actual execution result to optimize the collaborative execution of the unmanned station in a closed loop.

[0099] In step S150 of this embodiment, the execution process of updating the target action execution chain based on the execution error between the target execution result and the actual execution result to optimize the collaborative execution of the unmanned station in a closed loop includes: Step S151b: Obtain the return data of the execution unit and determine the actual execution result based on the return data; Step S152b: Calculate the execution error between the actual execution result and the target execution result, and update the weighted score of the selected target action execution chain based on the execution error; Step S153b: The update of the weighted score triggers the reselection of the target action execution chain, and the execution unit performs collaborative execution in the unmanned station through the reselected target action execution chain.

[0100] Feedback data includes, but is not limited to, action execution status, action execution delay, equipment operating status, area status changes, task completion status, and execution failure information. For example, access control equipment will send back the flow limiting execution status, inspection robots will send back on-site inspection image data, broadcasting equipment will send back the broadcast completion status, and so on. All the data sent back is feedback data from the execution unit.

[0101] The actual execution result is determined based on the obtained feedback data, and then the execution error is calculated based on the difference between the actual execution result and the target execution result. For example, the execution error may include action execution failure error, action execution delay error, resource conflict error, area state change error, and equipment operation abnormality error.

[0102] For example, if the population density in a region continues to rise after the regional flow restriction action is completed, it is determined that there is a regional control error in the current target action execution chain; and if the inspection robot fails to reach the designated area within the preset time, the feedback data indicates that there is an action execution delay, and the execution delay error is determined based on this action execution delay.

[0103] After obtaining the execution error, the weighted score of the selected target action execution chain is updated. It should be noted that the weighted score of the target action execution chain is used to characterize the adaptability and execution stability of the current action execution chain in the current operating scenario. In one embodiment, the weighted score, adapted to the selection of the target action execution chain, is calculated based on link response latency and link stability, with the corresponding link complexity and link response latency as residuals.

[0104] After updating the weighted score of the target action execution chain, a corresponding weighted score update triggers the reselection of the target action execution chain. Specifically, based on the currently updated weighted score and the weighted scores of other candidate action execution chains, a new target action execution chain that fits the current running state is selected.

[0105] At this point, it should be added that in the execution of step S140, in response to the station control event, after generating a set of candidate actions through knowledge graph matching and station operation context constraints, several candidate action execution chains are constructed through candidate action search and parsing. Then, the candidate action execution chain is selected by weighted scoring of each candidate action execution chain, and finally the selected candidate action execution chain is taken as the target action execution chain.

[0106] Therefore, the candidate action execution chains that were not selected in step S140, as well as the target action execution chains that were updated with weighted scores, will be used as the execution scope for reselecting target action execution chains. From these, target action execution chains that are suitable for the current target action execution chains will be reselected based on weighted scores. Furthermore, task conversion and adjustment of the execution scheduling sequence will be performed based on the station operation context information of the reselected target action execution chains.

[0107] For example, based on the current regional status, equipment operating status, and resource occupancy status, execution units can be reallocated and execution scheduling sequences can be adjusted.

[0108] This method allows for dynamic re-updating of the target action execution chain based on changes in the actual operating status of the unmanned station, thereby improving the collaborative control capabilities, action execution stability, and station operation adaptability in complex operating scenarios.

[0109] Ultimately, closed-loop optimization was achieved, and through this closed-loop optimization method, the dynamic adaptability and collaborative control stability of the unmanned station in complex operating environments were improved.

[0110] In summary, the embodiments of this application achieve real-time state understanding, dynamic collaborative decision-making, and closed-loop automated control in complex operating scenarios of unmanned stations through unified semantic modeling, structured event modeling, knowledge reasoning, and collaborative control of the target action execution chain. Compared with existing control methods based on fixed rules, the embodiments of this application can effectively improve the overall cognitive ability, collaborative control efficiency, and system operation stability in complex operating scenarios.

[0111] Therefore, regarding the implementation of intelligent control of unmanned stations in the embodiments of this application, the response to station control events, i.e., the generation of candidate action sets, is described in detail.

[0112] As mentioned above, event perception in the operation of unmanned stations is realized through event modeling, i.e. the construction of event objects. For the currently perceived station control events, it is necessary to determine the available candidate actions through knowledge graph matching and constraints of station operation context information. Then, several candidate collaborative action groups corresponding to the response event are obtained through action search. Finally, the target action execution chain is obtained by parsing the temporal dependency relationship and weighted scoring of each candidate collaborative action group.

[0113] See 6. Figure 6 It is based on Figure 1 The flowchart in the corresponding embodiment describes the execution process of generating a set of candidate actions for an event object through knowledge graph matching and constraints of station operation context information in step S140.

[0114] The execution process of generating a candidate action set for the event object through knowledge graph matching and station operation context information constraints in step S140 of this application embodiment includes: Step S141a: Match the event object in the knowledge graph and locate the starting node; Step S142a: Perform path reasoning using one-hop search and / or multi-hop search on the starting node to obtain at least one path from which an action can be found, and then extract candidate actions from at least one path. Step S143a: Perform constraint filtering on the candidate actions based on station operation context information to form a set of candidate actions for the event object. The constraint filtering includes executability constraints, risk constraints, and resource filtering constraints.

[0115] These steps will be explained in detail below.

[0116] Knowledge graphs are used to describe the relationships between equipment, areas, operating states, control events, and control actions during the operation of unmanned stations, in order to support dynamic control reasoning in complex operating scenarios.

[0117] Unlike general knowledge graphs, the knowledge graph in this embodiment is an operation control knowledge graph adapted to the unmanned station operation scenario. It not only includes semantic relationships between entities, but also control logic relationships, equipment linkage relationships, anomaly propagation relationships, and action dependency relationships during station operation.

[0118] The nodes in the knowledge graph include device nodes, region nodes, event nodes, state nodes, and action nodes; the edge relationships between nodes include device association relationships, region association relationships, event triggering relationships, action execution relationships, and temporal dependency relationships.

[0119] For example, in the established knowledge graph, there is a triggering relationship between the "equipment high temperature" event node and the "regional flow restriction" action node; there is a linkage relationship between the "regional congestion" event node and the "broadcast guidance" action node; and there is a temporal dependency relationship between the "equipment power failure" action node and the "backup equipment switching" action node.

[0120] In the knowledge graph matching process, the event object is first matched against the knowledge graph to locate the corresponding starting node. The starting node represents the initial inference node of the current station control event in the knowledge graph. Based on the state information and causal relationship fields in the event object, the corresponding event node or state node is searched in the knowledge graph, and the matched node is determined as the starting node. For example, when the event object contains state tags such as "equipment high temperature," "area personnel gathering," and "equipment abnormal alarm," the corresponding "equipment abnormal event" node is matched in the knowledge graph as the starting node.

[0121] In step S142a, path reasoning is used to search for control action paths associated with the current station control event in the knowledge graph. One-hop search refers to directly searching for action nodes that are related to the current starting node; multi-hop search refers to continuing to search for related nodes based on the current node to form a multi-hop control reasoning path.

[0122] By executing step S142a, at least one action reasoning path corresponding to the current station control event is obtained, and the corresponding action is further extracted from the at least one path to obtain candidate actions.

[0123] In step S143a, station operation context information is introduced to implement constraint filtering for candidate actions, so that the response to station control events can be adapted to the current operating environment. In other words, for the same station control event, the appropriate action should be different in different operating environments. For example, for a regional congestion event, regional flow restriction actions are applicable during peak hours, while broadcast guidance actions are applicable during off-peak hours.

[0124] The constraint filtering implemented by incorporating station operation context information, as mentioned earlier, includes at least executability constraints, risk constraints, and resource filtering constraints. Executability constraints determine whether the current candidate action has the actual conditions for execution; for example, when backup equipment is offline, the corresponding "backup equipment switching" action will be filtered. Risk constraints determine whether a candidate action may cause new operational risks; for example, in areas with high personnel density, directly executing the "equipment power off" action may increase the operational risk in the area, thus reducing the priority of the corresponding action or filtering it. Resource filtering constraints are used to determine whether the current station resources meet the conditions for action execution. For example, when all inspection robots are in the task execution state, the corresponding inspection action will be filtered out due to insufficient resources.

[0125] By implementing constraint filtering, the real-time operating status of the unmanned station can be combined with the selection of a set of candidate actions that are suitable for the current operating environment from multiple candidate actions obtained by knowledge graph reasoning, thereby improving the accuracy of subsequent target action execution chain generation and the ability to adapt to collaborative control.

[0126] See Figure 7 , Figure 7 It is based on Figure 1 The corresponding embodiment shows a flowchart describing the execution process of selecting the output target action execution chain from the candidate action set based on the correlation and temporal dependency between candidate extraordinary problems in step S141.

[0127] The execution process of selecting the output target action execution chain from the candidate action set based on the correlation and temporal dependency between candidate extraordinary problems in step S141 provided in this application embodiment includes: Step S141b: For candidate actions in the candidate action set, perform action group search based on the correlation and temporal dependency between candidate actions to obtain several candidate collaborative action groups for the event object. Step S142b: For each candidate collaborative action group, analyze the temporal dependencies of the candidate actions contained therein to determine the execution order of the actions and construct the candidate action execution chain; Step S143b: Perform multi-target selection of candidate action execution chains to obtain the target action execution chain.

[0128] By implementing action group search, it is possible to search for action combinations from the candidate action set to jointly complete the multi-execution unit collaborative task generated by the current station. Unlike traditional control methods that trigger a single action independently, this embodiment establishes collaborative control relationships between actions through action search, thereby achieving linkage control in complex operating scenarios.

[0129] Action group search is performed on all candidate actions in the candidate action set, based on the relationships and temporal dependencies between them. The relationships between candidate actions are used to describe whether there are control coordination relationships, functional complementarity relationships, and / or linkage execution relationships between different actions.

[0130] For example, there is a regional control coordination relationship between the "regional broadcast" action and the "regional flow restriction" action; there is a device linkage relationship between the "equipment power failure" action and the "backup equipment switching" action.

[0131] Correspondingly, timing dependency is used to describe the execution order of different candidate actions, such as the "standby device switching" action which needs to be executed after the "device power failure" action.

[0132] For example, for equipment malfunction events, several candidate collaborative action groups can be searched. Candidate collaborative action group one includes the "area broadcast" action, the "area flow restriction" action, and the "inspection confirmation" action; candidate collaborative action group two includes the "equipment power failure" action and the "backup equipment switching" action; candidate collaborative action group three includes the "area broadcast" action and the "equipment power failure" action.

[0133] After obtaining several candidate cooperative action groups, the temporal dependency of each candidate cooperative action group is resolved to determine the execution order of the candidate actions in each candidate cooperative action group and to construct the candidate action execution chain.

[0134] It should be understood that temporal dependency resolution is used to analyze the execution prerequisite relationship, execution blocking relationship, and parallel execution relationship between different candidate actions. The execution prerequisite relationship describes that a candidate action can only be executed after another candidate action has been executed; the execution blocking relationship describes that the execution of a candidate action will restrict the execution of other actions; and the parallel execution relationship describes that there is no strong temporal dependency between multiple candidate actions and they can be executed simultaneously.

[0135] Based on temporal dependencies, the action relationships in candidate collaborative action groups are analyzed, and the action execution order is established. Then, an action execution chain is constructed according to the action execution order.

[0136] In summary, this approach enables the embodiments of this application to generate candidate action execution chains that are adapted to the current unmanned station operation scenario based on the correlation and temporal dependencies between candidate actions.

[0137] After completing the construction of candidate action execution chains, multi-target selection is further performed on each candidate action execution chain to determine the target action execution chain that is suitable for the current unmanned station operation status from multiple autonomous action execution chains.

[0138] In the execution of this step S143b, multi-objective selection is used to consider the execution efficiency, operational risks, resource consumption, and collaborative stability of different candidate action execution chains under the current complex operating scenario from multiple dimensions, thereby avoiding the problem of insufficient control adaptation caused by the traditional single rule selection method.

[0139] In practice, based on the defined target dimension, corresponding link evaluation metrics are generated for each candidate action execution chain. These link evaluation metrics include, but are not limited to, link response latency, link execution success rate, link resource consumption, link operation risk, and link stability.

[0140] Link response latency describes the time required for the corresponding candidate action execution chain to complete from triggering; link success rate describes the probability of the corresponding candidate action execution chain successfully executing in historical operation; link resource occupancy describes the degree of resource occupancy of the corresponding candidate action execution chain on the current station; link operation risk describes the operation risks that the candidate action execution chain may cause during execution, such as the corresponding candidate action execution chain including the execution of the "equipment power failure" candidate action during peak operation, which may lead to an increase in regional operation risk.

[0141] Under the influence of the constructed link evaluation index, multi-objective selection of candidate action execution chains is implemented, and the selected candidate action execution chains are the target action execution chains.

[0142] In an exemplary embodiment, the multi-target selection of candidate action execution chains is based on link response latency and link stability indicators. Residual configuration is performed according to the corresponding link complexity and link response latency to obtain a weighted score execution process. Then, candidate action execution chains are selected based on the obtained weighted scores to obtain the target action execution chain.

[0143] Specifically, the execution process of step S143b includes: calculating the link response latency of the candidate action execution chain and estimating the link stability based on the action success rate; calculating a weighted score of the candidate action execution chain involving link response latency and link stability, configuring residuals based on the link complexity and link response latency corresponding to the candidate action execution chain in the weighted score calculation; selecting the candidate action execution chain based on the obtained weighted score, and using the selected candidate action execution chain as the target action execution chain.

[0144] In this exemplary embodiment, after obtaining multiple candidate action execution chains, each candidate action execution chain is evaluated, and the target action execution chain is selected based on this evaluation.

[0145] Unlike evaluating only a single action, this application embodiment takes the candidate action execution chain as the whole object. By comprehensively considering the response efficiency and execution stability of the candidate action execution chain, the target action execution chain is selected for the collaborative control scenario of unmanned station.

[0146] In the specific implementation, the link response latency of each candidate action execution chain is first calculated. Since a candidate action execution chain consists of multiple candidate actions, the link response latency is not only related to the execution time of a single candidate action, but also to the temporal dependencies and parallel execution relationships between candidate actions. For example, for a candidate action execution chain consisting of the candidate actions "area broadcast," "area flow limiting," and "inspection confirmation" in sequence, its link response latency is composed of the area broadcast latency, area flow limiting latency, and inspection confirmation latency. For a candidate action execution chain containing parallel execution nodes, such as the "area broadcast" candidate action and the "backup inspection equipment scheduling" candidate action being executed in parallel, and then the "inspection confirmation" candidate action being executed, the actual time consumed by the corresponding nodes is first determined based on the parallel execution relationship to obtain the overall link response latency.

[0147] In addition to link response latency, link stability is estimated for candidate action execution chains based on action success rate. In an exemplary embodiment, link stability is estimated based on the historical execution success rate of each candidate action in the candidate action execution chain. It should be noted that, in link stability estimation involving action success rate, action dependencies, device failure rate, and communication stability can also be incorporated to improve the accuracy of stability estimation.

[0148] To further elaborate, when there are many dependent candidate actions in a candidate action execution chain, the failure of any preceding action may prevent the execution of subsequent actions. Therefore, the stability of the chain is attenuated based on the action dependencies. For a candidate action execution chain, if all subsequent actions depend on the successful execution of preceding actions, then the corresponding chain stability will inevitably be lower than that of a candidate action execution chain where multiple actions can be executed independently.

[0149] Equipment failure rate reflects the probability of failure of the corresponding equipment during historical operation of the execution unit. The equipment failure rate is determined based on the historical failure records, maintenance records, or health status information of the execution unit, and the success rate of the corresponding action is adjusted according to the equipment failure rate.

[0150] Communication stability reflects the reliability of the communication link between the execution unit and the control platform. Communication stability is determined based on the number of communication interruptions, communication latency fluctuations, and network quality indicators, and the reliability of action execution is adjusted accordingly based on this communication stability.

[0151] By comprehensively considering action success rate, action dependency, equipment failure rate, and communication stability, the stability of candidate action execution chains can be estimated from multiple dimensions such as the reliability of the action itself, the reliability of the executing equipment, and the reliability of the execution environment. This improves the matching degree between the link stability assessment results and the actual operating scenario, and enhances the accuracy of target action execution chain selection.

[0152] After obtaining the link response latency and link stability, the weighted score of the candidate action execution chain is calculated. That is, the link response latency and link stability are used as the core evaluation indicators for weighted score calculation. At this time, the smaller the link response latency, the higher the corresponding weighted score; the higher the link stability, the higher the corresponding weighted score.

[0153] Furthermore, to avoid excessive bias towards candidate action execution chains containing simple or complex candidate actions, a link complexity and a residual corresponding to the link response delay (link response delay residual) are configured in the weighted scoring calculation process. Link complexity describes the number of actions, the number of action dependencies, and the number of cooperating execution nodes in the candidate action execution chain. The introduction of link complexity reflects the collaborative cost required for the execution of candidate actions to achieve the corresponding control objective, while the link response delay residual reflects the degree of deviation between the actual link response delay and the expected link response delay.

[0154] By introducing link complexity and link response delay residuals, the scoring bias caused by evaluating solely based on delay and stability is effectively avoided, thereby improving the matching degree between the evaluation results of candidate action execution chains and actual operating scenarios.

[0155] Subsequently, the candidate action execution chains will be selected based on the obtained weighted scores. Specifically, the candidate actions will be sorted according to their weighted scores to select the target action execution chain.

[0156] This exemplary embodiment enables the selection of a target action execution chain that balances response efficiency and execution stability from multiple candidate action execution chains. This improves the accuracy and reliability of collaborative control decisions in complex operating scenarios of unmanned stations and provides an optimized control link for subsequent collaborative execution by execution units.

[0157] Thus, the intelligent control process based on multimodal semantic understanding, event reasoning, and collaborative control in the unmanned station scenario has been realized.

[0158] However, in order to achieve the above-mentioned intelligent control of unmanned stations, as mentioned above, in the actual deployment process, it is also necessary to pre-build a cross-modal coding model that can realize the unified semantic expression of data from different modalities, and deploy the trained cross-modal coding model into the unmanned station intelligent control platform to support the mapping of video data, sensor data and text data to a unified semantic space during the operation phase.

[0159] It should be noted that, unlike publicly available datasets, the data generated by subway stations during long-term operation is usually from complex sources and contains a lot of noise. For example, video data may contain issues such as occlusion, changes in lighting, and abnormal images; sensor data may contain issues such as acquisition errors, equipment drift, and missing data; and text data may contain false alarms, duplicate alarms, and non-standard maintenance records.

[0160] Meanwhile, since different modal data are usually generated independently by different systems, data mismatch, time asynchrony, and inaccurate semantic annotation may occur during actual operation, inevitably resulting in a large number of noisy samples in the training data. If the above data is directly used to train a cross-modal coding model, it is easy for the model to learn incorrect cross-modal correspondences, thereby affecting the accuracy of subsequent unified semantic representation construction and the intelligent control effect of unmanned stations.

[0161] Therefore, in this embodiment of the application, before training the cross-modal coding model, noise samples are first identified and data is filtered for the multimodal training data generated during the operation of the subway station. After obtaining high-quality training samples, a cross-modal coding model that can adapt to the unmanned station scenario is constructed through cross-modal multi-layer alignment training, thereby improving the quality of unified semantic space construction and the state understanding and control decision-making capabilities in the subsequent intelligent control process.

[0162] The training process of the cross-modal coding model will be explained in detail below.

[0163] See Figure 8 , Figure 8 This is a flowchart illustrating an intelligent control method for unmanned stations based on multimodal semantic modeling, according to another exemplary embodiment.

[0164] The intelligent control method for unmanned stations based on multimodal semantic modeling provided in this application also includes: Step S210: For the prepared multimodal training data, the local features of each modality are identified by the modality-corresponding numberer. These local features correspond to local semantic units in the modality training data. Step S220: Based on the local features of local semantic units in the multimodal training data, construct training pairs across modalities, and calculate cross-modal similarity matrices for the training pairs based on local features. The cross-modal similarity matrix is ​​used to characterize the matching degree between two local semantic units across modalities. Training pairs with consistent semantics have a higher matching degree than training pairs with inconsistent semantics. Step S230: Construct an alignment optimization objective based on the cross-modal similarity matrix, and jointly train the cross-modal coding model through local feature alignment, bidirectional matching constraints and global semantic consistency constraints to obtain a cross-modal semantically aligned coding model.

[0165] As mentioned above, a unified semantic space is constructed for the intelligent control scenario of unmanned stations through the training phase of the embodiments of this application.

[0166] In this exemplary embodiment, the trained cross-modal coding model is deployed in step S120 and used to perform feature encoding and unified semantic mapping on real-time multimodal data, thereby generating a unified semantic representation used in the runtime phase.

[0167] In step S210, the multimodal training data comes from video data, equipment operation data, environmental monitoring data, alarm logs, and maintenance records accumulated during the long-term operation of subway stations.

[0168] Since different modalities of data have different representations, the data is first divided into local semantic units (LSUs) for each modality, and local features are extracted using the corresponding modality encoder. The LSU is used to represent the smallest semantic analysis unit in the modality data. For video modality data, video frames can be divided into multiple local regions (patches) using image segmentation, with each patch serving as a local semantic unit. For text modality data, word segmentation, phrase segmentation, or semantic fragment segmentation can be used to divide the text into multiple local semantic units. For sensor modality data, it can be divided according to time windows or state change windows to form multiple sensor local semantic units.

[0169] After dividing the data into local semantic units, feature extraction is performed on each of these units using a corresponding modal encoder. For example, a visual encoder extracts visual features corresponding to video blocks; a text encoder extracts text features corresponding to text semantic segments; and a sensor encoder extracts state features corresponding to time window state sequences. This yields the local feature representation of the corresponding local semantic unit. Although the data from each modality comes from different sources, they may all reflect the same abnormal state.

[0170] Therefore, local semantic units are used as the basis for cross-modal alignment, rather than directly based on the entire sample for alignment training. This allows the model to learn more fine-grained cross-modal correspondences and improves its ability to identify abnormal states in complex operating scenarios.

[0171] Furthermore, the local semantic unit alignment capability established during the training phase will be transferred to the unified semantic mapping process of real-time multimodal data in step S120 during the operation phase, enabling semantically consistent video blocks, text segments, and sensor state segments to obtain similar representations in the unified semantic space. This will improve the ability of the unified semantic representation to express abnormal state features and operation state features, providing a foundation for subsequent station control event recognition.

[0172] In step S220, after obtaining local features, cross-modal training pairs are further constructed. These training pairs describe whether different modalities express the same operational semantics. A cross-modal similarity matrix is ​​calculated based on the local features corresponding to the training pairs. The cross-modal similarity matrix characterizes the degree of matching between local semantic units of different modalities. Each element in the matrix corresponds to the similarity between two local semantic units. The higher the similarity, the higher the probability that they express the same operational semantics. From an operational perspective, the matching ability trained by the cross-modal similarity matrix will determine whether the features of different modalities in step S120 can be accurately mapped to the same semantic region in a unified semantic space.

[0173] In step S230, an alignment optimization objective is further constructed, and the cross-modal coding model is jointly trained through local feature alignment, bidirectional matching constraints, and global semantic consistency constraints. Local feature alignment ensures that semantically consistent local semantic units are brought closer together in a unified semantic space. For example, video features of high-temperature equipment and text features of abnormal equipment temperature are brought closer together, while irrelevant semantics are moved further apart. Bidirectional matching constraints simultaneously establish semantic matching capabilities between two modalities, such as simultaneously establishing video-to-text matching and text-to-video matching capabilities. Global semantic consistency constraints ensure that different modalities form a consistent overall semantic expression for the same operating scenario.

[0174] A cross-modal semantic alignment encoding model is obtained through joint optimization of local alignment, bidirectional matching, and global consistency constraints. After training, this encoding model is deployed to the runtime phase and used for unified semantic mapping of real-time multimodal data in step S120. The resulting encoding model, namely the aforementioned cross-modal encoding model that provides a unified semantic space, not only achieves semantic alignment between different modalities but also enables the unified semantic representation generated in the runtime phase to simultaneously possess fine-grained state recognition capabilities and cross-modal semantic association capabilities.

[0175] Furthermore, the execution process of step S220 includes: for local semantic units of training data of each modality, for all pairwise similarities of local features across modalities, constructing a cross-modal similarity matrix of cross-modal training pairs by calculating the obtained similarities, and training pairs located on the diagonal of the cross-modal similarity matrix are semantically consistent correct matching pairs.

[0176] Based on the local features of local semantic units in multimodal training data, cross-modal training pairs are constructed, and cross-modal similarity matrices are calculated for the training pairs based on local features.

[0177] Training pairs are used to represent the correspondence between two local semantic units from different modalities. By constructing training pairs oriented towards local semantic units in the current training batch, it is convenient to perform all pairwise similarity calculations on cross-modal local features, obtain the similarity corresponding to each training pair, and finally arrange them according to the correspondence of the training pairs to obtain the cross-modal similarity matrix.

[0178] The cross-modal similarity matrix is ​​used to characterize the overall matching relationship between local semantic units of different modalities in the current training batch. Each element in the cross-modal similarity matrix corresponds to the degree of matching of a training pair.

[0179] For example, the cross-modal similarity matrix obtained for training batches between video and text modalities is as follows:

[0180] in, Indicates the first i The video local semantic unit and the first j Similarity between local semantic units of a text. i and j The value ranges from 1 to N .

[0181] In multimodal training data, different modalities originate from the same operational scenario or the same event sample. Therefore, sample-level pairing relationships are established during the training data preparation phase. For example, the first video sample corresponds to the first text sample; the second video sample corresponds to the second text sample; ...; the Nth video sample corresponds to the Nth text sample. Therefore, the local semantic units obtained by partitioning corresponding samples naturally possess semantic consistency at the same index position. Based on this, training pairs located on the diagonal of the cross-modal similarity matrix can be determined as semantically consistent correct matching pairs, i.e.: , , ..., .

[0182] Other matching pairs correspond to incorrect or non-matching pairs. In other words, the diagonal elements in the cross-modal similarity matrix represent the matching relationships between cross-modal local semantic units from the same operating scenario, the same event sample, or the same state at the same time; the off-diagonal elements represent the matching relationships between different operating scenarios, different event samples, or different states. By constructing the cross-modal similarity matrix, this embodiment transforms the cross-modal semantic alignment problem into a problem of distinguishing between correct and non-matching pairs.

[0183] The local feature alignment, bidirectional matching constraints, and global semantic consistency constraints in subsequent steps S230 can all be constructed and optimized based on the cross-modal similarity matrix.

[0184] Furthermore, the cross-modal matching capability formed during the training phase will be transferred to the unified semantic mapping process in step S120 during the operation phase, enabling features expressing the same operational semantics in video blocks, text segments, and sensor status segments to aggregate in a unified semantic space, thereby improving the accuracy of unmanned station operation status recognition and event object construction.

[0185] In an exemplary embodiment, the execution process of step S230 includes: constructing matching labels for each local semantic unit in the current batch of multimodal training data, and calculating local alignment cross-entropy loss based on the cross-modal similarity matrix. The local alignment cross-entropy loss is used to make the local features of the correct matching pairs closer to each other and the local features of the incorrect matching pairs farther apart during the backpropagation process of the cross-modal coding model, thereby completing cross-modal semantic alignment in the local sample space constructed by the current batch of multimodal training data.

[0186] In this embodiment, local alignment is used to establish fine-grained semantic correspondences between local semantic units of different modalities, so that local semantic units expressing the same operational semantics can obtain similar representations in a unified semantic space.

[0187] Since step S220 has already obtained the cross-modal similarity matrix corresponding to the current training batch, the pre-existing sample pairing relationships in the further training data are used to construct matching labels. Specifically, in this embodiment, the different modal training samples in the current batch come from the same running scenario or the same event sample. Therefore, the local semantic units obtained by dividing the corresponding samples can be regarded as correct matching pairs at the same index position.

[0188] Based on this, training pairs corresponding to the diagonal positions of the cross-modal similarity matrix are labeled as correctly matched pairs, and corresponding matching labels are constructed. These labels indicate the correct matching positions, i.e., the pairs... This indicates that the i-th position is the correct match.

[0189] After obtaining the matching labels, the local alignment cross-entropy loss is calculated based on the cross-modal similarity matrix. Specifically, the similarity in the cross-modal similarity matrix is ​​first normalized to obtain the cross-modal matching probability distribution corresponding to the current local semantic unit. A higher matching probability indicates a higher probability that the two corresponding local semantic units express the same operational semantics. Then, the local alignment cross-entropy loss is calculated using the constructed matching labels and the matching probability distribution.

[0190] Local alignment cross-entropy loss measures the difference between the current cross-modal coding model output and the correct matching relationship. The loss decreases when the correct matching pair has a higher matching probability, and increases when the incorrect matching pair has a higher matching probability. Therefore, during model training, local alignment cross-entropy loss continuously guides the model to learn the correct cross-modal correspondences.

[0191] Furthermore, during backpropagation, the system updates the parameters of the cross-modal coding model based on the local alignment cross-entropy loss. For correctly matched pairs, the cross-modal coding model gradually increases their similarity, thereby bringing the corresponding local features closer together in the unified semantic space. For incorrectly matched pairs, the cross-modal coding model gradually decreases their similarity, thereby moving the corresponding local features further apart in the unified semantic space.

[0192] After multiple training batches and iterations, a stable cross-modal semantic correspondence can be gradually established within the local sample space constructed in the current batch, achieving cross-modal semantic alignment at the local semantic unit level.

[0193] Furthermore, the local semantic aggregation capability formed through local alignment training will be transferred to the unified semantic mapping process in step S120 during the operation phase, so that features expressing the same operating state in real-time video blocks, text segments, and sensor state segments can be mapped to similar regions in the unified semantic space.

[0194] This improves the ability of the unified semantic representation to express operational status characteristics such as abnormal equipment status, regional congestion status, and environmental risk status, providing a more accurate semantic foundation for subsequent station control event identification, event object construction, and target action execution chain generation.

[0195] To further elaborate, the aforementioned execution process of constructing matching labels for each local semantic unit in the current batch of multimodal training data and calculating the local alignment cross-entropy loss based on the cross-modal similarity matrix includes: Based on the data pairing relationships existing in the multimodal data, the training pairs distributed on the diagonal of the cross-modal similarity matrix are identified as the correct matching pairs; Labels are constructed based on the location distribution of correct matching pairs. These labels are used to indicate the index position of the local semantic units contained in the correct matching pair in the modal data.

[0196] Since the training data has already established data pairing relationships between cross-modal samples during the data preparation phase, there are corresponding relationships between different modal samples in the current batch.

[0197] For example, the first video sample and the first text sample in the current batch correspond to the same running scenario, the second video sample and the second text sample correspond to the same running scenario, and so on.

[0198] Therefore, after constructing the cross-modal similarity matrix based on the current batch of samples, the training pairs corresponding to the same index position constitute the correct matching pairs and are distributed in the diagonal position of the cross-modal similarity matrix.

[0199] Matching labels are constructed based on the positional distribution of correct matching pairs in the cross-modal similarity matrix. These matching labels record the index information corresponding to the position of the correct matching pair, indicating the target matching object corresponding to the current local semantic unit. For example, for the i-th video local semantic unit, if its correct matching object is the i-th text local semantic unit, then the system uses index position i as the corresponding matching label.

[0200] Furthermore, for each local semantic unit in the current batch, a corresponding matching label can be constructed in the manner described above, thereby forming a label set corresponding to the training data of the current batch.

[0201] The above method eliminates the need for additional manual annotation of matching relationships between local semantic units. Instead, it automatically generates matching labels by directly utilizing the existing data pairing relationships in multimodal data, thereby providing supervision information for subsequent calculation of local alignment loss based on cross-modal similarity matrices.

[0202] Furthermore, since the constructed labels directly correspond to the positional distribution of the correct matching pairs in the cross-modal similarity matrix, they can guide the cross-modal coding model to accurately learn the correspondence between local semantic units of different modalities during training, thereby improving the cross-modal semantic alignment effect.

[0203] In another embodiment, and further specifically, the aforementioned execution process of constructing matching labels for each local semantic unit in the current batch of multimodal training data and calculating the local alignment cross-entropy loss based on the cross-modal similarity matrix includes: The similarity probability distribution between local semantic units is calculated based on the cross-modal similarity matrix, and the local alignment cross-entropy loss is calculated based on the similarity probability distribution to complete the cross-modal alignment loss calculation of local semantic units at the local alignment level.

[0204] First, a similarity probability distribution is calculated based on the similarity values ​​corresponding to each local semantic unit in the cross-modal similarity matrix. The similarity probability distribution is used to characterize the probability that the current local semantic unit forms a correct matching relationship with each local semantic unit in the cross-modal matrix.

[0205] For example, for a given video local semantic unit, a row of similarity values ​​in its corresponding cross-modal similarity matrix represents the degree of matching between that video local semantic unit and each text local semantic unit. After normalizing these similarity values, the similarity probability distribution of the corresponding video local semantic unit with respect to all text local semantic units is obtained. At this point, if the current video local semantic unit has a high similarity to the correctly matching text local semantic unit, the matching probability at the corresponding position is relatively high; conversely, if it has a high similarity to non-matching text local semantic units, the matching probability at the corresponding position increases.

[0206] After obtaining the similarity probability distribution, the local alignment cross-entropy loss is further calculated by combining the constructed matching labels.

[0207] The local alignment cross-entropy loss measures the difference between the current similarity probability distribution and the target matching relationship. When a correctly matched pair obtains a high matching probability at its corresponding position, the local alignment cross-entropy loss decreases; conversely, when a non-matching pair obtains a high matching probability at its corresponding position, the local alignment cross-entropy loss increases. Therefore, the local alignment cross-entropy loss reflects the learning effect of the current cross-modal coding model on local semantic unit matching relationships.

[0208] During model training, backpropagation is performed based on the calculated local alignment cross-entropy loss to update the cross-modal coding model parameters, so that the similarity between the corresponding local features of the correctly matched pairs gradually increases, while the similarity between the corresponding local features of the unmatched pairs gradually decreases.

[0209] After continuous iterative training, semantically consistent local semantic units will gradually cluster into similar regions in the unified semantic space, while semantically inconsistent local semantic units will gradually separate, thereby establishing a stable cross-modal semantic correspondence at the local alignment level.

[0210] Furthermore, the local semantic aggregation capability formed by the local alignment cross-entropy loss can improve the semantic consistency of different modal data when mapping to a unified semantic space during the operation phase. This enables features expressing the same operating state in video blocks, text segments, and sensor state segments to achieve a higher degree of aggregation, thereby improving the ability of the unified semantic representation to express the operating state of the unmanned station and providing a more accurate semantic foundation for subsequent event state recognition and event object construction.

[0211] See Figure 9 , Figure 9 It is based on Figure 8 The corresponding embodiment shows a flowchart describing the steps of constructing an alignment optimization objective based on a cross-modal similarity matrix, and jointly training a cross-modal coding model through local feature alignment, bidirectional matching constraints, and global semantic consistency constraints to obtain a cross-modal semantically aligned coding model.

[0212] The embodiment of this application provides a step S230: constructing an alignment optimization objective based on a cross-modal similarity matrix, and jointly training a cross-modal coding model through local feature alignment, bidirectional matching constraints, and global semantic consistency constraints to obtain a cross-modal semantically aligned coding model. This step includes: Step S231: To train bidirectional matching for two modalities, the similarity probability of each matching direction is calculated based on the cross-modal similarity matrix, and the bidirectional matching loss corresponding to bidirectional matching of the training pair is calculated based on the similarity probability of each matching direction; and Step S232: Construct global semantic consistency constraints based on the global semantic representation of the current batch of multimodal training data, and calculate global consistency loss based on the distance between each modality and the global semantic representation; Step S233: Construct a joint optimization objective based on local alignment cross-entropy loss, bidirectional matching loss and global consistency loss, and iteratively train the cross-modal coding model based on the joint optimization objective; In the iterative training, the parameters of the cross-modal coding model are updated through backpropagation so that the local features of the correct matching pairs are closer to each other, and the local features of the incorrect matching pairs are farther apart, thereby achieving cross-modal semantic alignment.

[0213] In step S231, when establishing a unidirectional matching relationship solely through local alignment cross-entropy loss, a semantic learning imbalance between different modalities can easily occur. For example, the video modality can accurately match the text modality, but the matching ability decreases when the text modality retrieves the video modality in reverse. Therefore, it is necessary to introduce bidirectional matching constraints.

[0214] In practice, for any training pair formed between two modalities, the similarity probability distributions corresponding to the two matching directions are calculated based on the cross-modal similarity matrix. For example, between the video modality and the text modality, the first matching direction is used to describe the probability distribution of local semantic units in the video matching local semantic units in the text; the second matching direction is used to describe the probability distribution of local semantic units in the text matching local semantic units in the video.

[0215] Then, based on the matching results in the two directions, the corresponding matching loss is calculated, and the bidirectional matching loss is further obtained.

[0216] This approach not only requires the video to find the correct text, but also the text to find the correct video, thereby establishing a more stable cross-modal bidirectional semantic mapping relationship. In step S232, the alignment between local semantic units can only guarantee fine-grained semantic consistency, but cannot guarantee that different modalities form a unified overall semantic understanding of the same running scenario. Therefore, it is necessary to introduce global semantic consistency constraints.

[0217] In the specific implementation, a corresponding global semantic representation is first constructed based on the current batch of multimodal training data. The global semantic representation is used to characterize the overall semantic information corresponding to the current running scenario or the current event sample.

[0218] For example, for the same device malfunction scenario: the video modality shows a device malfunction image; the text modality shows a device malfunction alarm; and the sensor modality shows a device temperature rise. Although the local semantic units are different, they all reflect the same device malfunction event. Therefore, it is possible to construct a corresponding global semantic representation based on the features of each modality.

[0219] Then, the semantic distance between each modality feature and the global semantic representation is calculated, and the global consistency loss is calculated based on the semantic distance.

[0220] In this way, different modalities are aligned not only at the local semantic level, but also form a unified semantic expression at the scene level. Furthermore, the global consistency constraint can improve the overall expressive power of the unified semantic representation in the runtime phase for complex runtime scenarios, enabling the event state recognition in the subsequent step S130 to simultaneously utilize multiple modalities to form a unified state cognition.

[0221] In step S233, a joint optimization objective is constructed. Specifically, the local alignment cross-entropy loss constrains fine-grained semantic alignment between local semantic units; the bidirectional matching loss constrains bidirectional semantic correspondence between different modalities; and the global consistency loss constrains the unified semantic expression of different modalities for the overall operating scenario. Therefore, these three types of losses jointly optimize the cross-modal coding model at the local, cross-modal, and scenario levels, respectively.

[0222] Then, iterative training of the cross-modal coding model is performed based on the joint optimization objective. During training, the parameters of the cross-modal coding model are continuously updated through backpropagation, so that the local features corresponding to correct matches gradually move closer, while the local features corresponding to incorrect matches gradually move further away. At the same time, different modalities form a more consistent overall semantic representation for the same operating scenario. After multiple rounds of training, a cross-modal semantically aligned cross-modal coding model is obtained.

[0223] After training, the cross-modal coding model is deployed to the runtime phase and used in the unified semantic mapping process in step S120, enabling real-time video data, text data, and sensor data to be stably mapped to the unified semantic space.

[0224] This improves the accuracy and stability of unmanned station operation status recognition, event object construction, knowledge graph reasoning, and target action execution chain generation.

[0225] See Figure 10 , Figure 10 This is a flowchart illustrating an intelligent control method for unmanned stations based on multimodal semantic modeling, according to another embodiment.

[0226] The intelligent control method for unmanned stations based on multimodal semantic modeling provided in this application also includes: Step S310: Calculate the loss value for each of the current batch of multimodal sample data, and use the calculated loss value to select target training samples; Step S320: Noise is filtered by performing dual-network collaborative learning on the target training samples to obtain denoised multimodal training data, which is used for multi-layer alignment training.

[0227] Because the video, sensor, and text data generated during the operation of unmanned stations come from different business systems, the training data often contains noisy samples. For example, video footage may be obstructed, blurry, or abnormally captured; sensor data may be drifted, distorted, or missing; text data may contain false alarms, duplicate alarms, and errors in manual recording; and there may also be time synchronization issues or sample mismatches between different modalities.

[0228] If the aforementioned noisy samples are directly used in cross-modal semantic alignment training, it can easily lead to the model learning incorrect cross-modal correspondences, thus affecting the quality of subsequent unified semantic space construction. Therefore, before performing local alignment, bidirectional matching, and global consistency training, noise screening is first performed on the current batch of multimodal sample data.

[0229] In the specific implementation, the training loss value for each multimodal sample is calculated based on the current cross-modal coding model. The training loss value reflects the degree of matching between the corresponding sample and the semantic rules learned by the current model. Generally, samples with correct semantic correspondences are easier for the model to learn, and their corresponding loss values ​​are relatively low; while samples with noise, mismatches, or anomalous annotations are more difficult for the model to fit correctly, and their corresponding loss values ​​are usually higher. Therefore, the loss value can serve as a basis for evaluating sample quality.

[0230] Then, target training samples are selected based on the loss value distribution corresponding to the current batch of samples. These target training samples represent data samples with high confidence in the current batch. This method allows for the priority retention of training samples that conform to the true cross-modal correspondences during training.

[0231] Furthermore, to reduce the potential bias that a single model might introduce when selecting samples, a dual-network collaborative learning approach is employed for noise filtering. Specifically, a first network and a second network are constructed that are independent of each other. The first network and the second network share the same training objective but have different parameter initializations, resulting in different sample learning paths during their training process.

[0232] The first network selects low-loss samples in the current batch based on its own loss distribution; the second network similarly selects corresponding low-loss samples based on its own loss distribution.

[0233] Because different networks have varying sensitivities to noisy samples, two networks typically will not reach consistent judgments on the same noisy samples at the same stage. Therefore, a collaborative learning mechanism is employed, whereby low-loss samples selected by the first network are used to guide the training of the second network, and vice versa.

[0234] The above cross-update method avoids the problem of a single network continuously reinforcing its own misjudged samples.

[0235] Furthermore, the samples selected by the two networks are fused to obtain denoised multimodal training data. This denoised training data retains high-confidence samples that are commonly or partially recognized by different networks, thus improving sample quality while maintaining the scale of the training samples. After obtaining the denoised training data, local feature alignment training, bidirectional matching training, and global semantic consistency training are performed based on it. Because the data used in the training has already undergone noise filtering, the correct matching relationships in the cross-modal similarity matrix are more accurate, and the correspondences between local semantic units are more reliable.

[0236] Furthermore, the bidirectional matching loss and global consistency loss can also be based on more accurate cross-modal correspondences, thereby improving the training stability of cross-modal coding models.

[0237] Ultimately, the trained cross-modal coding model, when performing unified semantic mapping during the runtime phase, can more accurately identify the semantic relationships between video data, sensor data, and text data, improve the quality of unified semantic representation, and thus improve the accuracy of station control event recognition, knowledge graph reasoning, and target action execution chain generation.

[0238] In an exemplary embodiment, the execution process of step S310 includes: calculating the loss value of each sample data in the current batch of multimodal sample data; and determining the clean data in the current batch of multimodal sample data as the target training samples by fitting the calculated loss value to the distribution of GMM.

[0239] Specifically, the cross-modal coding model of the current training epoch is used to perform forward computation on each multimodal sample in the current batch, and the loss value of the corresponding sample is obtained. The loss value is used to characterize the degree of matching between the current sample and the semantic rules learned by the model.

[0240] During cross-modal training, samples with correct semantic correspondences are usually easier for the model to learn, and therefore their loss values ​​are usually smaller; while samples with modality mismatch, labeling errors, abnormal collection, or semantic noise are more difficult for the model to fit correctly, and therefore their corresponding loss values ​​are usually larger.

[0241] After obtaining the loss values ​​for all samples in the current batch, the distribution of loss values ​​is modeled and analyzed. It should be noted that the distribution of loss values ​​typically changes dynamically across different training stages and different batches of data, making it difficult to accurately distinguish between clean and noisy samples using a fixed loss threshold. For example, in the early stages of training, even correct samples may have high loss values; while in the later stages of training, the loss values ​​of some noisy samples may decrease.

[0242] Therefore, this embodiment uses a Gaussian Mixture Model (GMM) to fit the probability distribution of the loss value of the current batch of samples.

[0243] Using the loss values ​​of all samples in the current batch as input, the distribution of loss values ​​is clustered and modeled using a Generative Model (GMM). The GMM can divide the current loss value distribution into multiple probability distribution components, where: the distribution component with lower loss values ​​corresponds to data samples that are easier for the model to learn; and the distribution component with higher loss values ​​corresponds to data samples that may contain noise.

[0244] The confidence level of a sample is determined by the probability that each sample belongs to a low-loss distribution component. A higher probability that a sample belongs to a low-loss distribution component indicates that the corresponding sample is more likely to be a clean sample; conversely, a lower probability indicates that the corresponding sample is more likely to be a noisy sample.

[0245] High-confidence samples are selected from the current batch of samples to serve as target training samples. For example, samples with correct semantic correspondence between video, text, and sensor data typically have loss values ​​clustered in low-loss regions and are therefore classified as clean data; while samples with misaligned timestamps, incorrect event labeling, or abnormal sensor acquisition typically have loss values ​​located in high-loss regions and are therefore identified as noisy data. In this way, target training samples can be dynamically determined based on the current training state without the need to pre-set a fixed selection threshold.

[0246] Furthermore, since GMM utilizes the overall distribution characteristics of the loss values ​​of the current batch of samples, it can adapt to changes in the distribution of loss values ​​at different training stages, thereby improving the accuracy of clean sample identification.

[0247] After obtaining the target training samples, subsequent dual-network collaborative learning and noise filtering processes are performed based on the target training samples, thereby providing a higher quality training data foundation for cross-modal local alignment training, bidirectional matching training and global semantic consistency training.

[0248] Therefore, it is possible to dynamically evaluate and screen the quality of training samples in the case of modality mismatch, abnormal acquisition and noise labeling of multimodal data in unmanned stations, thereby improving the training stability of cross-modal coding models and the quality of unified semantic space construction.

[0249] In another exemplary embodiment, the execution process of step S320 includes: using the initialized first network and the second network, respectively, to screen low-loss samples of the target training samples obtained in the current batch, to obtain the low-loss samples screened by the first network and the low-loss samples screened by the second network; and fusing the low-loss samples to obtain denoised multimodal training data (taking the union).

[0250] After obtaining the target training samples in step S310, the target training samples are further filtered for noise using a dual-network collaborative learning mechanism.

[0251] It should be noted that even after filtering by loss value distribution, the target training samples may still contain some latent noise samples. For example, when there are slight temporal misalignments, semantic biases, or weak annotation errors between some modalities, their loss values ​​may be close to those of normal samples, thus not being completely eliminated. If such samples are directly used for subsequent cross-modal alignment training, they may still interfere with the model learning process. Therefore, this embodiment further constructs a first network and a second network to form a dual-network collaborative learning structure. The first network and the second network have the same network structure, but are trained using different parameter initialization methods, so that the two networks form different parameter distributions and sample learning paths in the early stages of training.

[0252] Because different initialization parameters lead to differences in the network's focus on sample features, the judgment results of two networks for noisy samples are usually not completely consistent. Specifically, the first network performs forward computation based on the target training samples of the current batch and sorts the samples according to their corresponding loss values, selecting low-loss samples that the first network considers to have high credibility. Simultaneously, the second network performs the same process and obtains its corresponding low-loss samples. Low-loss samples represent samples that the current network can easily learn and that conform to the current semantic rules. Compared to noisy samples, low-loss samples typically have higher cross-modal semantic consistency and more accurate sample pairing relationships.

[0253] In this embodiment, after the first network and the second network independently complete the screening of low-loss samples, the screening results of the two networks are fused.

[0254] Specifically, the low-loss sample set selected by the first network and the low-loss sample set selected by the second network are combined to obtain the final denoised multimodal training data. The reason for using the union fusion method is that when a sample is identified as a reliable sample by either network, it is retained for subsequent training, thus avoiding the problem of effective samples being prematurely discarded due to misjudgment by a single network. For example, if the first network can correctly identify an abnormal sample of a device, but the second network fails to identify the sample due to differences in training stages, the sample can still be retained for subsequent training through union fusion. Compared to only retaining the intersection of samples jointly selected by the two networks, union fusion can retain more effective training data while ensuring sample quality, thereby maintaining the diversity and scene coverage of the training data.

[0255] Furthermore, during subsequent training, low-loss samples selected by the first network can be used to guide the parameter updates of the second network, and vice versa. This cross-learning approach reduces the risk of a single network continuously memorizing noisy samples, allowing both networks to gradually learn more stable and consistent cross-modal semantic correspondences. After obtaining denoised multimodal training data, local feature alignment training, bidirectional matching training, and global semantic consistency training are performed using this data. Since the training data has undergone dual-network collaborative filtering, the correct matching relationships in the cross-modal similarity matrix are more accurate, and the correspondences between local semantic units are more stable, thereby improving the convergence stability and semantic alignment accuracy of the cross-modal coding model.

[0256] Furthermore, once the trained cross-modal coding model is deployed to the runtime stage, it can improve the accuracy of semantic mapping between video blocks, text segments, and sensor state segments, making the semantic aggregation effect in the unified semantic space more obvious, thereby improving the accuracy of subsequent event state recognition, event object construction, and target action execution chain generation.

[0257] Through the above methods, this embodiment not only utilizes the loss distribution to screen target training samples, but also further filters hidden noise samples through dual-network collaborative learning, thereby providing a high-quality training data foundation for subsequent multi-layer semantic alignment training and improving the generalization ability and stability of the cross-modal coding model in the complex operation scenario of unmanned stations.

[0258] In one exemplary embodiment, this application also provides a computer device including a memory, a processor, and a computer program stored in the memory, the processor executing the computer program to implement the steps of the method as described above.

[0259] In one exemplary embodiment, this application also provides a computer program product including a computer program that, when executed by a processor, implements the steps of the method as described above.

[0260] In one exemplary embodiment, this application also provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the method as described above.

[0261] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this application.

[0262] In an exemplary embodiment of this application, a computer-readable storage medium is also provided, on which computer-readable instructions are stored, which, when executed by a computer's processor, cause the computer to perform the methods described in the above method embodiments.

[0263] According to one embodiment of this application, a program product for implementing the methods in the above-described method embodiments is also provided. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and can run on a terminal device, such as a personal computer. However, the program product of this invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0264] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0265] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0266] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, or any suitable combination thereof.

[0267] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0268] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0269] Furthermore, although the steps of the method in this application are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0270] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, and includes several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0271] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the appended claims.

Claims

1. An intelligent control method for unmanned stations based on multimodal semantic modeling, characterized in that, The method includes: Collect real-time multimodal data from unmanned stations, wherein the real-time multimodal data includes at least video data, sensor data, and text data; By mapping real-time multimodal data to a unified semantic space, a unified semantic representation is obtained. The unified semantic space is constructed through multi-level alignment training across modalities, which includes local alignment, bidirectional alignment, and global alignment. Based on the unified semantic representation, an event object is constructed for the unmanned station that currently triggers a station control event. The event object includes at least state information, causal relationship fields, and impact information. The construction steps of the event object include: extracting the current abnormal state features and operating state features of the unmanned station through event state recognition using unified semantic representation to obtain at least one initial event category; generating state labels for the initial event category based on multiple state attributes corresponding to the obtained at least one initial event category, as well as quantization parameter values ​​for the state labels; and constructing a structured event object for the unmanned station that currently triggers a station control event by using the state labels and their quantization parameter values, comprising state information, causal relationship fields, and impact information. For the event object, a candidate action set is generated through knowledge graph matching and station operation context information constraints. Based on the correlation and temporal dependency between candidate actions, an output target action execution chain is selected from the candidate action set. The steps for generating the candidate action set include: matching the event object in the knowledge graph to locate the starting node; performing one-hop search and / or multi-hop search path reasoning on the starting node to obtain at least one path that can find the action, and then extracting candidate actions from the at least one path; and performing station operation context information constraint filtering on the candidate actions to form the candidate action set of the event object. The constraint filtering includes executability constraints, risk constraints, and resource filtering constraints. The execution unit implements the coordinated execution of the unmanned station through the target action execution chain, and updates the target action execution chain again according to the execution error between the target execution result and the actual execution result, so as to optimize the coordinated execution of the unmanned station in a closed loop.

2. The method according to claim 1, characterized in that, The process of mapping real-time multimodal data to a unified semantic space to obtain a unified semantic representation includes: In response to the input of real-time multimodal data during the operation phase, feature encoding is performed on each modal data in the real-time multimodal data to obtain the corresponding modal features; The modal features are mapped to a unified semantic space constructed through cross-modal multi-layer alignment training to generate a unified semantic representation with cross-modal semantic consistency. The mapping step of the modal features to the unified semantic space includes using a cross-modal coding model optimized through local alignment, bidirectional alignment and global alignment during the training phase to project different modal features to the unified semantic space, so that semantically consistent modal features will have similar representations in the unified semantic space.

3. The method according to claim 1, characterized in that, The method further includes: Given the prepared multimodal training data, the local features of each modality are extracted from the modal training data by the encoder corresponding to each modality. The local features correspond to local semantic units in the modal training data. Based on the local features of local semantic units in multimodal training data, training pairs are constructed across modalities, and a cross-modal similarity matrix is ​​calculated for the training pairs based on local features. The cross-modal similarity matrix is ​​used to characterize the degree of matching between two local semantic units across modalities. Training pairs with consistent semantics have a higher degree of matching than training pairs with inconsistent semantics. Based on the cross-modal similarity matrix, an alignment optimization objective is constructed, and the cross-modal coding model is jointly trained through local feature alignment, bidirectional matching constraints, and global semantic consistency constraints to obtain a cross-modal semantically aligned coding model.

4. The method according to claim 3, characterized in that, The process involves constructing an alignment optimization objective based on the cross-modal similarity matrix, and jointly training the cross-modal coding model through local feature alignment, bidirectional matching constraints, and global semantic consistency constraints to obtain a cross-modal semantically aligned coding model, including: To train bidirectional matching for two modalities, the similarity probability of each matching direction is calculated based on the cross-modal similarity matrix, and the bidirectional matching loss corresponding to bidirectional matching of the training pair is calculated based on the similarity probability of each matching direction; and A global semantic consistency constraint is constructed based on the global semantic representation of the current batch of multimodal training data, and the global consistency loss is calculated based on the distance between each modality and the global semantic representation. A joint optimization objective is constructed based on the local alignment cross-entropy loss, the bidirectional matching loss, and the global consistency loss, and the cross-modal coding model is iteratively trained based on the joint optimization objective; In the iterative training, the parameters of the cross-modal coding model are updated through backpropagation so that the local features of the correct matching pairs are closer to each other, and the local features of the incorrect matching pairs are farther apart, thereby achieving cross-modal semantic alignment.

5. The method according to claim 1, characterized in that, The method further includes: The loss value is calculated for each batch of multimodal sample data, and the target training samples are selected based on the calculated loss value. By performing dual-network collaborative learning to filter noise on the target training samples, denoised multimodal training data is obtained, which is then used for multi-layer alignment training.

6. The method according to claim 5, characterized in that, The step of calculating loss values ​​for each of the current batch of multimodal sample data and using the calculated loss values ​​to select target training samples includes: Calculate the loss value for each sample in the current batch of multimodal sample data; The calculated loss value is fitted to a distribution using a Gaussian Mixture Model (GMM) to determine the clean data in the current batch of multimodal sample data as the target training samples.

7. The method according to claim 1, characterized in that, The step of selecting the output target action execution chain from the candidate action set based on the correlation and temporal dependency between candidate actions includes: Multi-index feature extraction is performed on each candidate action in the candidate action set to obtain candidate action features; For candidate actions in the candidate action set, an action group search is performed based on the correlation and temporal dependency between the candidate actions to obtain several candidate collaborative action groups for the event object; For each candidate collaborative action group, the temporal dependencies of the candidate actions contained therein are analyzed to determine the execution order of the actions and construct the candidate action execution chain; Perform multi-target selection on the candidate action execution chain to obtain the target action execution chain.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-7.

Images