Port machine shadow driving data acquisition and man-machine bifurcation triggering method and port machine shadow driving data acquisition and man-machine bifurcation triggering system
By detecting the differences between the shadow model and human operation in real time in the port machinery system, collecting multimodal data and determining the degree of human-machine divergence, the problem of recognition error and decision deviation in the port machinery assisted driving system under complex working conditions is solved, thereby improving the stability and safety of the system and increasing the efficiency of data acquisition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BROAD VISION (XIAMEN) TECHNOLOGY CO LTD
- Filing Date
- 2026-02-02
- Publication Date
- 2026-05-05
AI Technical Summary
Existing intelligent assisted driving systems for port machinery suffer from identification errors and decision-making biases under complex operating conditions, making it difficult to guarantee the stability and safety of the system. Furthermore, they have low data acquisition efficiency, high costs, and lack closed-loop optimization mechanisms.
By detecting the differences between the shadow model and human operation in real time under manual driving conditions, multimodal auxiliary judgment data is collected, human-machine divergence index is calculated, divergence triggering events are determined, and relevant data is used for retraining of the shadow model to build a continuous learning and closed-loop optimization system.
This improved the stability and safety of the port machinery assisted driving system under complex working conditions, increased data acquisition efficiency, reduced data filtering and labeling costs, and enabled dynamic optimization of the model.
Smart Images
Figure CN121980285A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of mechanical intelligence technology, and in particular to a method and system for data acquisition and human-machine divergence triggering of port machinery shadow driving. Background Technology
[0002] With the development of port automation and intelligence, port machinery (including quay cranes, yard cranes, rubber-tired gantry cranes, reach stackers, and forklifts) is gradually introducing assisted driving systems based on visual perception and deep learning during loading and unloading operations. Existing intelligent assisted driving systems for port machinery mainly rely on the fusion of multiple cameras, radar, IMUs, and spreader sensors to achieve functions such as target detection, obstacle avoidance warning, trajectory planning, and spreader attitude control.
[0003] However, due to limitations in the coverage of the training dataset and the diversity of environments, the model still suffers from recognition errors and decision-making biases under complex working conditions such as strong light, rain and fog, nighttime, swaying of the spreader, or reflections from special containers, making it difficult to guarantee the stability and safety of the system.
[0004] In existing technologies, improving model accuracy typically relies on offline acquisition and manual annotation of large amounts of video data. However, this approach has the following problems: (1) The data acquisition efficiency is low and it is difficult to cover abnormal scenarios and critical states of port machinery in real operating environments; (2) Data filtering and labeling are costly and it is difficult to automatically identify differences between model misjudgment and human operation; (3) The model update lacks a closed-loop mechanism and cannot achieve dynamic optimization of "learning in use". Summary of the Invention
[0005] This application provides a method and system for collecting port machinery shadow driving data and triggering human-machine divergence, which can detect the difference between the shadow model and human operation in real time under manual driving conditions and obtain retraining data for the shadow model based on the difference, thereby constructing a continuous learning and closed-loop optimization system for port machinery assisted driving models.
[0006] The above-mentioned objective of this application is achieved through the following technical solution: Firstly, this application provides a method for collecting data on shadow driving of port machinery and triggering human-machine divergence, including: The system collects the first instruction set of the manual control system and the multimodal auxiliary judgment data associated with the first instruction set. The multimodal auxiliary judgment data includes visual information, three-dimensional point cloud information, spreader status parameters, port machinery attitude information, and environmental information. The multimodal auxiliary judgment data is input into the shadow model and the second instruction set output by the shadow model is obtained; The human-machine divergence index is obtained by calculating the first instruction set and the second instruction set; The divergence triggering event is determined based on the human-machine divergence index, and the divergence type and risk level of the divergence triggering event are recorded simultaneously. If the conditions for determining a divergence trigger event are met within consecutive frames, the event corresponding to the consecutive frames is marked as a consecutive divergence trigger event. The shadow model and the manual driving control system operate in parallel, and the second instruction set output by the shadow model is not given to the port crane.
[0007] In one possible implementation of the first aspect, the calculation of the human-machine divergence index for the first instruction set and the second instruction set includes: sampling time t i The first instruction set of the artificial control system is denoted as A. h (t) i The second instruction set of the shadow model is denoted as A. m (t) i ); Calculate the difference vector between the first instruction set and the second instruction set: ∆A(t i ) = A h (t) i )- A m (t) i ); Let the weighted coefficient vector be:
[0008] Weighted normalization of the difference vector yields the human-machine divergence index: .
[0009] In one possible implementation of the first aspect, determining the divergence trigger event based on the human-machine divergence index includes: Set a divergence threshold The trigger determination function is defined as follows:
[0010] In the formula, This indicates that the event has been triggered. Branching events also include consecutive branching events, the criteria for determining consecutive branching events are as follows:
[0011] In the formula, The length of the continuous frame window. To determine the threshold.
[0012] In one possible implementation of the first aspect, the first instruction set of the manual control system includes: Determine the content to be collected: Establish a unified timeline: Synchronize the collected content with the corresponding control variables: In the formula, the sampling frequency is Sampling period .
[0013] In one possible implementation of the first aspect, after determining the divergence triggering event, the method further includes caching the multimodal auxiliary judgment data associated with the divergence triggering event. Caching includes: Create a cache time window:
[0014] Determine the corresponding multimodal auxiliary judgment data based on the cache time window:
[0015] In the formula, Pre-caching duration before triggering Extended duration after triggering; Cache the multimodal auxiliary judgment data in a circular buffer. middle And indexed by timestamp and event level.
[0016] In one possible implementation of the first aspect, the multimodal auxiliary judgment data is cached in a circular buffer. The process also includes generating event description tuples and data packets from cached multimodal auxiliary judgment data and event levels; The cached multimodal auxiliary judgment data and event level are used to generate an event description tuple as follows:
[0017] In the formula, Types of divergence; Risk level; : Context; The generated data packet is as follows:
[0018] In the formula, Encode() represents compression and indexing operations.
[0019] In one possible implementation of the first aspect, the generated data packet is further sent to the cloud and a shadow model in the cloud is trained. Training the shadow model in the cloud includes: The cloud uses the received data packets Constructing a set of difficult examples:
[0020] Identifying high-frequency scenes using clustering algorithms:
[0021] Add typical hard examples of each class to the retraining set. ; Periodically update the shadow model: .
[0022] Secondly, this application provides a port machinery shadow driving data acquisition and human-machine divergence triggering device, including: The information acquisition unit is used to acquire the first instruction set of the manual control system and the multimodal auxiliary judgment data associated with the first instruction set. The multimodal auxiliary judgment data includes visual information, three-dimensional point cloud information, spreader status parameters, port machinery attitude information and environmental information. The information input unit is used to input multimodal auxiliary judgment data into the shadow model and obtain the second instruction set output by the shadow model; The computing and processing unit is used to calculate the human-machine divergence index obtained from the first instruction set and the second instruction set. The event processing unit is used to determine the divergence-triggered event based on the human-machine divergence index and simultaneously record the divergence type and risk level of the divergence-triggered event. If the conditions for determining a divergence trigger event are met within consecutive frames, the event corresponding to the consecutive frames is marked as a consecutive divergence trigger event. The shadow model and the manual driving control system operate in parallel, and the second instruction set output by the shadow model is not given to the port crane.
[0023] Thirdly, this application provides a port machinery shadow driving data acquisition and human-machine divergence triggering system, the system comprising: One or more memories for storing instructions; and One or more processors are configured to call and execute the instructions from the memory to perform the methods described in the first aspect and any possible implementation thereof.
[0024] Fourthly, this application provides a computer-readable storage medium, the computer-readable storage medium comprising: The program, when run by a processor, is executed as described in the first aspect and any possible implementation thereof.
[0025] Fifthly, this application provides a computer program product, including program instructions that, when run by a computing device, execute the method described in the first aspect and any possible implementation thereof.
[0026] Sixthly, this application provides a chip system including a processor for implementing the functions involved in the foregoing aspects, such as generating, receiving, transmitting, or processing the data and / or information involved in the foregoing methods.
[0027] This chip system can consist of chips or include chips and other discrete components.
[0028] In one possible design, the chip system also includes a memory for storing necessary program instructions and data. The processor and the memory can be decoupled and located on different devices, connected via wired or wireless means, or the processor and the memory can be coupled to the same device. Attached Figure Description
[0029] Figure 1 This is the overall method flowchart provided in this application.
[0030] Figure 2 This is a diagram of a port machinery shadow driving system architecture provided in this application.
[0031] Figure 3 This is a schematic diagram of the human-machine divergence calculation principle provided in this application. Detailed Implementation
[0032] The technical solutions in this application will be further described in detail below with reference to the accompanying drawings.
[0033] This application discloses a method for data acquisition and human-machine divergence triggering in port machinery shadow driving. Please refer to [link / reference]. Figure 1 In some examples, the port crane shadow driving data collection and human-machine divergence triggering method disclosed in this application includes the following steps: S101, collects the first instruction set of the manual control system and the multimodal auxiliary judgment data associated with the first instruction set. The multimodal auxiliary judgment data includes visual information, three-dimensional point cloud information, spreader status parameters, port machinery attitude information and environmental information. S102, input the multimodal auxiliary judgment data into the shadow model and obtain the second instruction set output by the shadow model; S103, calculate the human-machine divergence index of the first instruction set and the second instruction set; S104, determine the divergence triggering event based on the human-machine divergence index and simultaneously record the divergence type and risk level of the divergence triggering event; If the conditions for determining a divergence trigger event are met within consecutive frames, the event corresponding to the consecutive frames is marked as a consecutive divergence trigger event. The shadow model and the manual driving control system operate in parallel, and the second instruction set output by the shadow model is not given to the port crane.
[0034] It should be noted that the shadow model involved in the technical solution of this application is directly deployed to the port machinery assisted driving platform. The port machinery assisted driving platform and the manual driving control system operate synchronously. The manual driving control system is controlled by the operator, and the instructions given by the operator are sent to the port machinery. The instructions given by the shadow model are only used as outputs and are not sent to the port machinery.
[0035] Here, we first need to define the input multimodal signal vector of the port machinery system, as follows:
[0036] in: • Visual information (a collection of image frames from multiple cameras); • Radar / point cloud features (3D point cloud information); • : Lifting gear status parameters (angle, position, load); • : Port aircraft attitude information (GPS coordinates, IMU data); • Environmental information (wind speed, light intensity, humidity, etc.).
[0037] The shadow model is defined as a non-interventional prediction function:
[0038] in: The virtual control commands predicted by the model. These are the model parameters.
[0039] The model is running in read-only mode:
[0040] That is, the predicted signal does not enter the actual execution channel. .
[0041] For example, suppose in Extract 6 features:
[0042] in The distance to the obstacle is (m). The lateral deviation is (m). Relative velocity (m / s), Swing angle (°), Wind speed (m / s), Height (m). Let... The shadow model at this layer is approximated as:
[0043] Substitute the values:
[0044] Therefore
[0045] Should It does not enter the execution channel; it is only used for recording and comparison.
[0046] In step S101, the first instruction set of the manual control system and the multimodal auxiliary judgment data associated with the first instruction set are first collected. The multimodal auxiliary judgment data includes visual information, three-dimensional point cloud information, spreader status parameters, port machinery attitude information and environmental information.
[0047] The first instruction set of a manual control system refers to the specific instructions issued by the operator. There may be one or more instructions, which are collectively referred to as the first instruction set.
[0048] The multimodal auxiliary judgment data associated with the first instruction set refers to the basis for the generation of the first instruction set by the operator.
[0049] The first instruction set of the manual control system includes: Determine the content to be collected: Establish a unified timeline: Synchronize the collected content with the corresponding control variables: In the formula, the sampling frequency is Sampling period .
[0050] For example: Pick ,but . from Record 5 sampling points:
[0051] exist Record manual control (example: 5-dimensional control variables):
[0052] and input with the same timestamp Binding: .
[0053] In step S102, the multimodal auxiliary judgment data is input into the shadow model and the second instruction set output by the shadow model is obtained. Then, in step S103, the human-machine divergence index is calculated by the first instruction set and the second instruction set.
[0054] Specifically, the multimodal auxiliary judgment data given to the manual control system will generate a first instruction set, while the data given to the shadow model will generate a second instruction set. If the first instruction set and the second instruction set are the same, it means that the output of the shadow model is correct at this time. If the first instruction set and the second instruction set are different, it means that the output of the shadow model has an error or is incorrect at this time.
[0055] In step S104, the divergence triggering event will be determined based on the human-machine divergence index, and the divergence type and risk level of the divergence triggering event will be recorded simultaneously. The human-machine divergence index here has only one criterion: the degree of difference between the first instruction set and the second instruction set. The calculation method for the human-machine divergence index is as follows: sampling time t i The first instruction set of the artificial control system is denoted as A. h (t) i The second instruction set of the shadow model is denoted as A. m (t) i ); Calculate the difference vector between the first instruction set and the second instruction set: ∆A(t i ) = A h (t) i )- A m (t) i ); Let the weighted coefficient vector be:
[0056] Weighted normalization of the difference vector yields the human-machine divergence index: .
[0057] For example: exist ,
[0058] Therefore
[0059] Take the weights (example, sum to 1):
[0060] but
[0061] If the conditions for determining a branching event are met within consecutive frames, the event corresponding to the consecutive frames is marked as a consecutive branching event. In some cases, the disagreement-triggered events determined by the human-machine disagreement index include: Set a divergence threshold The trigger determination function is defined as follows:
[0062] In the formula, This indicates that the event has been triggered. Branching events also include consecutive branching events, the criteria for determining consecutive branching events are as follows:
[0063] In the formula, The length of the continuous frame window. To determine the threshold.
[0064] For example: set up 5 frames (sampling points) were obtained
[0065] but
[0066] If we take the length of the continuous window (i.e., 3-frame window) and , to the window :
[0067] Therefore, it can be determined that a persistent divergence event exists (in which...). (Starting from).
[0068] In some examples, after determining the divergence triggering event, the process also includes caching the multimodal auxiliary judgment data associated with the divergence triggering event. Caching includes: Create a cache time window:
[0069] Determine the corresponding multimodal auxiliary judgment data based on the cache time window:
[0070] In the formula, Pre-caching duration before triggering Extended duration after triggering; Cache the multimodal auxiliary judgment data in a circular buffer. middle And indexed by timestamp and event level.
[0071] For example: Get the trigger point ,make , Then the cache time window:
[0072] like Window duration Then the number of time series samples:
[0073] If the video from the 4 cameras is Number of buffered frames per path:
[0074] In some cases, multimodal auxiliary decision data is cached in a circular buffer. The process also includes generating event description tuples and data packets from cached multimodal auxiliary judgment data and event levels; The cached multimodal auxiliary judgment data and event level are used to generate an event description tuple as follows:
[0075] In the formula, Types of divergence; Risk level; : Context; The generated data packet is as follows:
[0076] In the formula, Encode() represents compression and indexing operations.
[0077] For example: set up , ,
[0078] but
[0079] If the bitrate of each video stream is approximately (H.265), Window The approximate size of each video stream is:
[0080] By overlaying the sensing / control timing and index files, we obtain... .
[0081] The data package needs to be uploaded. The specific steps are as follows: Define the upload priority function:
[0082] Where (B) represents the currently available bandwidth; Upload requirements:
[0083] That is, upload is performed when the overall priority is greater than the threshold; Packet queue:
[0084] Upload securely in chronological and hierarchical order.
[0085] For example: Convert risk levels into numerical values (example mapping): Low ,middle ,high . Pick threshold .
[0086] Event A: High risk, average bandwidth
[0087] therefore .
[0088] Event B: Low risk, ample bandwidth
[0089] Therefore, it will not be entered into the immediate upload queue.
[0090] The uploaded data package is used to train the shadow model in the cloud. The specific method for training the shadow model in the cloud is as follows: The cloud uses the received data packets Constructing a set of difficult examples:
[0091] Identifying high-frequency scenes using clustering algorithms:
[0092] Add typical hard examples of each class to the retraining set. ; Periodically update the shadow model: .
[0093] For example: Suppose the cloud receives 6 event fragments ,
[0094] To visually demonstrate clustering, two statistical features are taken for each event (example):
[0095] And we got 6 points:
[0096] Clustering (such as DBSCAN / k-means, etc.) can yield example clusters:
[0097] and These may be outliers / noise points (depending on the algorithm's threshold). High-frequency / high-risk clusters (such as...) The corresponding data segments are added to the retraining set. And perform parameter updates:
[0098] This application also provides a port machinery shadow driving data acquisition and human-machine divergence triggering device, including: The information acquisition unit is used to acquire the first instruction set of the manual control system and the multimodal auxiliary judgment data associated with the first instruction set. The multimodal auxiliary judgment data includes visual information, three-dimensional point cloud information, spreader status parameters, port machinery attitude information and environmental information. The information input unit is used to input multimodal auxiliary judgment data into the shadow model and obtain the second instruction set output by the shadow model; The computing and processing unit is used to calculate the human-machine divergence index obtained from the first instruction set and the second instruction set. The event processing unit is used to determine the divergence-triggered event based on the human-machine divergence index and simultaneously record the divergence type and risk level of the divergence-triggered event. If the conditions for determining a divergence trigger event are met within consecutive frames, the event corresponding to the consecutive frames is marked as a consecutive divergence trigger event. The shadow model and the manual driving control system operate in parallel, and the second instruction set output by the shadow model is not given to the port crane.
[0099] Furthermore, the calculation of the human-machine divergence index for the first instruction set and the second instruction set includes: sampling time t i The first instruction set of the artificial control system is denoted as A. h (t) i The second instruction set of the shadow model is denoted as A. m (t) i ); Calculate the difference vector between the first instruction set and the second instruction set: ∆A(t i ) = A h (t) i )- A m (t) i ); Let the weighted coefficient vector be:
[0100] Weighted normalization of the difference vector yields the human-machine divergence index: .
[0101] Furthermore, based on the human-machine divergence index, divergence-triggered events include: Set a divergence threshold The trigger determination function is defined as follows:
[0102] In the formula, This indicates that the event has been triggered. Branching events also include consecutive branching events, the criteria for determining consecutive branching events are as follows:
[0103] In the formula, The length of the continuous frame window. To determine the threshold.
[0104] Furthermore, the first instruction set of the manual control system includes: Determine the content to be collected: Establish a unified timeline: Synchronize the collected content with the corresponding control variables: In the formula, the sampling frequency is Sampling period .
[0105] Furthermore, after determining the divergence triggering event, the process also includes caching the multimodal auxiliary judgment data associated with the divergence triggering event. Caching includes: Create a cache time window:
[0106] Determine the corresponding multimodal auxiliary judgment data based on the cache time window:
[0107] In the formula, Pre-caching duration before triggering Extended duration after triggering; Cache the multimodal auxiliary judgment data in a circular buffer. middle And indexed by timestamp and event level.
[0108] Furthermore, the multimodal auxiliary judgment data is cached in a circular buffer. The process also includes generating event description tuples and data packets from cached multimodal auxiliary judgment data and event levels; The cached multimodal auxiliary judgment data and event level are used to generate an event description tuple as follows:
[0109] In the formula, Types of divergence; Risk level; : Context; The generated data packet is as follows:
[0110] In the formula, Encode() represents compression and indexing operations.
[0111] Furthermore, it also includes sending the generated data packets to the cloud and training the shadow model in the cloud. Training the shadow model in the cloud includes: The cloud uses the received data packets Constructing a set of difficult examples:
[0112] Identifying high-frequency scenes using clustering algorithms:
[0113] Add typical hard examples of each class to the retraining set. ; Periodically update the shadow model: .
[0114] In one example, the unit in any of the above devices may be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), or one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs), or a combination of at least two of these integrated circuit forms.
[0115] For example, when the units in the device can be implemented through a processing element scheduler, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor capable of calling programs. Alternatively, these units can be integrated together to form a system-on-a-chip (SOC).
[0116] In this application, various objects such as messages / information / devices / network elements / systems / apparatus / actions / operations / processes / concepts may be named. It is understood that these specific names do not constitute a limitation on the relevant objects. The names may be changed depending on the scenario, context, or usage habits. The understanding of the technical meaning of the technical terms in this application should be mainly determined from their functions and technical effects embodied / performed in the technical solution.
[0117] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0118] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0119] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0120] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0121] It should also be understood that in the various embodiments of this application, the terms "first," "second," etc., are merely to indicate that multiple objects are different. For example, a first time window and a second time window are only to indicate different time windows. They should not have any effect on the time windows themselves, and the aforementioned terms "first," "second," etc., should not impose any limitations on the embodiments of this application.
[0122] It should also be understood that, in the various embodiments of this application, unless otherwise specified or in case of logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referenced by each other, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships.
[0123] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a computer-readable storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned computer-readable storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0124] This application also provides a port machinery shadow driving data acquisition and human-machine divergence triggering system, the system comprising: One or more memories for storing instructions; and One or more processors are configured to retrieve and execute the instructions from the memory, performing the methods described above.
[0125] This application also provides a computer program product including instructions that, when executed, cause the terminal device and the network device to perform operations corresponding to the methods described above.
[0126] This application also provides a chip system including a processor for implementing the functions involved in the above description, such as generating, receiving, transmitting, or processing the data and / or information involved in the above methods.
[0127] This chip system can consist of chips or include chips and other discrete components.
[0128] The processor mentioned above can be a CPU, a microprocessor, an ASIC, or one or more integrated circuits that execute a program to control the method of transmitting the feedback information described above.
[0129] In one possible design, the chip system also includes a memory for storing necessary program instructions and data. The processor and the memory can be decoupled and located on different devices, connected via wired or wireless means to support the chip system in implementing the various functions described in the above embodiments. Alternatively, the processor and the memory can also be coupled to the same device.
[0130] Optionally, the computer instructions are stored in memory.
[0131] Optionally, the memory can be a storage unit within the chip, such as a register or cache. Alternatively, the memory can be a storage unit located outside the chip within the terminal, such as a ROM or other types of static storage devices that can store static information and instructions, such as RAM.
[0132] It is understood that the memory in this application may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory.
[0133] Non-volatile memory can be ROM, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory.
[0134] Volatile memory can be RAM, which is used as an external cache. There are many different types of RAM, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory.
[0135] The embodiments described in this specific implementation are preferred embodiments of this application and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.
Claims
1. A method for data acquisition and human-machine divergence triggering in port crane shadow driving, characterized in that, include: The system collects the first instruction set of the manual control system and the multimodal auxiliary judgment data associated with the first instruction set. The multimodal auxiliary judgment data includes visual information, three-dimensional point cloud information, spreader status parameters, port machinery attitude information, and environmental information. The multimodal auxiliary judgment data is input into the shadow model and the second instruction set output by the shadow model is obtained; The human-machine divergence index is obtained by calculating the first instruction set and the second instruction set; The divergence triggering event is determined based on the human-machine divergence index, and the divergence type and risk level of the divergence triggering event are recorded simultaneously. If the conditions for determining a divergence trigger event are met within consecutive frames, the event corresponding to the consecutive frames is marked as a consecutive divergence trigger event. The shadow model and the manual driving control system operate in parallel, and the second instruction set output by the shadow model is not given to the port crane.
2. The method for collecting port crane shadow driving data and triggering human-machine divergence according to claim 1, characterized in that, The human-machine divergence index obtained by calculating the first instruction set and the second instruction set includes: sampling time t i The first instruction set of the artificial control system is denoted as A. h (t) i The second instruction set of the shadow model is denoted as A. m (t) i ); Calculate the difference vector between the first instruction set and the second instruction set: ∆A(t i ) = A h (t i )- A m (t i ); Let the weighted coefficient vector be: ； Weighted normalization of the difference vector yields the human-machine divergence index: 。 3. The method for collecting port crane shadow driving data and triggering human-machine divergence according to claim 2, characterized in that, According to the human-machine divergence index, divergence-triggered events include: Set a divergence threshold The trigger determination function is defined as follows: ； In the formula, This indicates that the event has been triggered. Branching events also include consecutive branching events, the criteria for determining consecutive branching events are as follows: ； In the formula, The length of the continuous frame window. To determine the threshold.
4. The method for collecting port crane shadow driving data and triggering human-machine divergence according to claim 1, characterized in that, The first instruction set of the manual control system includes: Determine the content to be collected: ; Establish a unified timeline: ; Synchronize the collected content with the corresponding control variables: ; In the formula, the sampling frequency is Sampling period .
5. The method for collecting port machinery shadow driving data and triggering human-machine divergence according to claim 1, characterized in that, After determining the divergence trigger event, the process also includes caching the multimodal auxiliary judgment data associated with the divergence trigger event. Caching includes: Create a cache time window: ； Determine the corresponding multimodal auxiliary judgment data based on the cache time window: ； In the formula, Pre-caching duration before triggering Extended duration after triggering; Cache the multimodal auxiliary judgment data in a circular buffer. middle And indexed by timestamp and event level.
6. The method for collecting port crane shadow driving data and triggering human-machine divergence according to claim 5, characterized in that, Cache the multimodal auxiliary judgment data in a circular buffer. The process also includes generating event description tuples and data packets from cached multimodal auxiliary judgment data and event levels; The cached multimodal auxiliary judgment data and event level are used to generate an event description tuple as follows: ； In the formula, Types of divergence; Risk level; : Context; The generated data packet is as follows: ； In the formula, Encode() represents compression and indexing operations.
7. The method for collecting port machinery shadow driving data and triggering human-machine divergence according to claim 6, characterized in that, It also includes sending the generated data packets to the cloud and training the shadow model in the cloud. Training the shadow model in the cloud includes: The cloud uses the received data packets Constructing a set of difficult examples: ； Identifying high-frequency scenes using clustering algorithms: ； Add typical hard examples of each class to the retraining set. ; Periodically update the shadow model: 。 8. A port machinery shadow driving data acquisition and human-machine divergence triggering device, characterized in that, include: The information acquisition unit is used to acquire the first instruction set of the manual control system and the multimodal auxiliary judgment data associated with the first instruction set. The multimodal auxiliary judgment data includes visual information, three-dimensional point cloud information, spreader status parameters, port machinery attitude information and environmental information. The information input unit is used to input multimodal auxiliary judgment data into the shadow model and obtain the second instruction set output by the shadow model; The computing and processing unit is used to calculate the human-machine divergence index obtained from the first instruction set and the second instruction set. The event processing unit is used to determine the divergence-triggered event based on the human-machine divergence index and simultaneously record the divergence type and risk level of the divergence-triggered event. If the conditions for determining a divergence trigger event are met within consecutive frames, the event corresponding to the consecutive frames is marked as a consecutive divergence trigger event. The shadow model and the manual driving control system operate in parallel, and the second instruction set output by the shadow model is not given to the port crane.
9. A port machinery shadow driving data acquisition and human-machine divergence triggering system, characterized in that, The system includes: One or more memories for storing instructions; and One or more processors are configured to retrieve and execute the instructions from the memory to perform the method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes: The program, when run by a processor, executes the method as described in any one of claims 1 to 7.