Method and apparatus for retrieving job behavior data

Abstract

The present disclosure provides a method and device for retrieving work behavior data, and relates to the fields of artificial intelligence and smart mines. The method for retrieving work behavior includes: receiving input information of a target object; performing intent understanding on the input information to obtain a behavior event intent, the behavior event intent indicating data retrieval of a target behavior event related to work behavior performed by the work object; determining a target work feature from a plurality of preset work features based on the behavior event intent, the plurality of preset work features corresponding to a plurality of work behavior data of multiple modalities, the target modal reference work feature including a reference sub-feature associated with a preset behavior event, and the preset sub-features of the plurality of preset work features being associated based on time attributes or event attributes, the target work feature including a target sub-feature matching the target behavior event; and pushing target work behavior data determined based on the target work feature to the target object.

Description

Technical Field

[0001] This disclosure relates to the fields of artificial intelligence and smart mining, and more specifically, to a method and apparatus for retrieving operational behavior data. Background Technology

[0002] With the rapid development of technology, equipment in mining and other operational scenarios can collect data through inertial measurement units, lidar, cameras, and other data acquisition devices. Based on this data, it can perform tasks such as autonomous driving, unloading mineral materials, and terrain surveying. Furthermore, the data collected by the equipment can be retrieved to optimize the operational control system, improve algorithm models, or perform operational event analysis, thereby meeting the actual needs for optimizing operational efficiency.

[0003] In realizing the concept disclosed herein, the inventors discovered at least the following problems in the related technologies: the data retrieval efficiency for work scenarios is low, requiring a lot of interactive operations and retrieval time, making it difficult to meet the actual needs of optimizing work efficiency. Summary of the Invention

[0004] In view of this, the present disclosure provides a method and apparatus for retrieving job behavior data.

[0005] One aspect of this disclosure provides a method for retrieving job behavior data, comprising: receiving input information from a target object; performing intent understanding on the input information to obtain a behavior event intent, the behavior event intent representing a target behavior event related to a job behavior performed by the job object; based on the behavior event intent, determining a target job feature from a plurality of preset job features characterizing the job behavior, the plurality of preset job features corresponding to a plurality of multimodal job behavior data, wherein the baseline job features of the target modality among the plurality of preset job features include a baseline sub-feature associated with a preset behavior event, the preset sub-features of each of the plurality of preset job features being associated based on a time attribute or an event attribute, the target job feature including a target sub-feature matching the target behavior event; and pushing the target job behavior data determined based on the target job feature to the target object.

[0006] Another aspect of this disclosure provides an apparatus for retrieving job behavior data, comprising: a receiving module for receiving input information from a target object; an intent understanding module for performing intent understanding on the input information to obtain a behavior event intent, wherein the behavior event intent represents a target behavior event related to a job behavior performed by the job object; a determining module for determining a target job feature based on the behavior event intent from a plurality of preset job features characterizing the job behavior, wherein the plurality of preset job features correspond to a plurality of multimodal job behavior data, wherein the baseline job feature of the target modality among the plurality of preset job features includes a baseline sub-feature associated with a preset behavior event, and the preset sub-features of each of the plurality of preset job features are associated based on a time attribute or an event attribute, and the target job feature includes a target sub-feature matching the target behavior event; and a pushing module for pushing the target job behavior data determined based on the target job feature to the target object.

[0007] Another aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the method as described above.

[0008] Another aspect of this disclosure provides a computer-readable storage medium storing computer-executable instructions, which, when executed, are used to implement the method described above.

[0009] Another aspect of this disclosure provides a computer program product including computer-executable instructions that, when executed, implement the method described above.

[0010] According to embodiments of this disclosure, by performing intent understanding on input information of any modality input by the target object, and determining the target object's intent to retrieve target behavioral events related to operational behavior, the data attributes of the operational behavior data that the target object needs to obtain are understood more accurately, thereby improving retrieval accuracy based on the behavioral event intent. By determining target sub-features from baseline operational features with a specified modality based on the behavioral event intent, and by determining target sub-features of other modalities from preset operational features of other modalities different from the specified modality based on the temporal attribute correlation between multiple preset sub-features, multi-modal target sub-features are accurately retrieved as target operational features according to the target object's behavioral event intent. This allows for the push of target operational behavior data determined based on the target operational features to the target object to meet its data retrieval needs, improving data retrieval efficiency and accuracy. This, in turn, improves the efficiency of data retrieval for operational environments such as mining operations, improves the execution efficiency of specific tasks such as mining operation algorithm updates and terrain data updates, and improves the analysis efficiency of abnormal operational states such as abnormal operational behaviors, thereby achieving the technical effect of improving the stability and safety of mining operations. Attached Figure Description

[0011] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments of this disclosure with reference to the accompanying drawings, in which:

[0012] Figure 1 The illustration schematically shows an exemplary system architecture of a method and apparatus for retrieving job behavior data according to embodiments of the present disclosure.

[0013] Figure 2 A flowchart illustrating a method for retrieving job behavior data according to an embodiment of this disclosure is shown schematically.

[0014] Figure 3 A schematic diagram illustrating the preset operation features according to an embodiment of the present disclosure is shown.

[0015] Figure 4 The illustration shows an application scenario of a method for retrieving job behavior data according to an embodiment of the present disclosure.

[0016] Figure 5 A block diagram of an apparatus for retrieving job behavior data according to an embodiment of the present disclosure is shown schematically.

[0017] Figure 6 A block diagram of an electronic device suitable for implementing the method for retrieving job behavior data described above, according to an embodiment of the present disclosure, is shown schematically. Detailed Implementation

[0018] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0019] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0020] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0021] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0022] In the embodiments disclosed herein, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of data (e.g., including but not limited to user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures have been taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0023] In the embodiments disclosed herein, user authorization or consent is obtained before acquiring or collecting user personal information.

[0024] In the process of retrieving data collected in the working environment of mining operations, the diverse and complex nature of data requirements, coupled with the high data quality requirements and diverse data types for specific data needs such as autonomous driving model training or evaluation, leads to low retrieval efficiency for operational behavior data collected by equipment. It is difficult to accurately and quickly obtain data that meets actual needs, thus resulting in low efficiency in optimizing operational processes.

[0025] This disclosure provides a method and apparatus for retrieving job behavior data. The method includes: receiving input information from a target object; performing intent understanding on the input information to obtain a behavior event intent, whereby the behavior event intent represents a target behavior event related to a job behavior performed by the job object; based on the behavior event intent, determining a target job feature from multiple preset job features characterizing the job behavior, wherein the multiple preset job features correspond to multiple job behavior data of multiple modalities, and the baseline job features of the target modality among the multiple preset job features include baseline sub-features associated with preset behavior events, and the preset sub-features of each of the multiple preset job features are associated based on time attributes or event attributes, and the target job feature includes target sub-features matching the target behavior event; and pushing the target job behavior data determined based on the target job feature to the target object.

[0026] Figure 1 This illustration schematically depicts an exemplary system architecture for a method and apparatus for retrieving job behavior data according to embodiments of this disclosure. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0027] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0028] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social media platform software, etc. (for example only).

[0029] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0030] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0031] It should be noted that the method for retrieving job behavior data provided in this embodiment can generally be executed by server 105. Correspondingly, the apparatus for retrieving job behavior data provided in this embodiment can generally be located in server 105. The method for retrieving job behavior data provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the apparatus for retrieving job behavior data provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Alternatively, the method for retrieving job behavior data provided in this embodiment can also be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103, or by other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103. Accordingly, the device for retrieving job behavior data provided in this embodiment of the present disclosure may also be installed in the first terminal device 101, the second terminal device 102 or the third terminal device 103, or in other terminal devices different from the first terminal device 101, the second terminal device 102 or the third terminal device 103.

[0032] It should be understood that Figure 1The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0033] Figure 2 A flowchart illustrating a method for retrieving job behavior data according to an embodiment of this disclosure is shown schematically.

[0034] like Figure 2 As shown, the method for retrieving job behavior data includes operations S210 to S240.

[0035] In operation S210, input information from the target object is received.

[0036] In operation S220, the input information is interpreted to obtain the intent of the behavioral event.

[0037] In operation S230, based on the intent of the behavioral event, the target task feature is determined from multiple preset task features that characterize the task behavior.

[0038] In operation S240, target operation behavior data determined based on target operation characteristics is pushed to the target object.

[0039] According to embodiments of this disclosure, the input information may include information in any modality, such as voice, text, or images. The target object can input the corresponding input information by performing interactive operations.

[0040] According to embodiments of this disclosure, the behavioral event is intended to represent data retrieval of target behavioral events related to the work behavior performed by a work object. The work object may include excavators, mining trucks, water trucks, etc., performing work tasks in a work scenario. The work behavior performed by the work object may include any type of work task such as unloaded driving, heavy-load driving, and excavating mineral materials. Embodiments of this disclosure do not limit the specific types of work behaviors and work objects.

[0041] According to embodiments of this disclosure, behavioral events can represent attribute information related to a work behavior, such as the name, attributes, and abnormal states of the work behavior. In some embodiments, behavioral events corresponding to attributes related to the work behavior can be determined based on preset rules. Behavioral events may include autonomous driving events, or abnormal work events, work behaviors related to a specified work area, etc.

[0042] In some embodiments, large models can be used to understand the intent of input information and obtain the intent of behavioral events. Large models can be deep learning models with a large parameter scale and complex computational structure; for example, large models may include large language models (LLMs). Large models can process input information based on a large number of model parameters to perform intent recognition and determine the intent of behavioral events.

[0043] In some embodiments, the input information can also be processed by interacting with the target object through dialogue based on an agent, thereby enabling the agent to understand the behavioral event intent represented by the input information of the target object.

[0044] According to embodiments of this disclosure, multiple preset task features correspond to multiple task behavior data in multiple modalities. For example, task audio features correspond to task audio data in an audio modality. Task audio features can be determined by encoding the task audio data. As another example, task image features can correspond to task images acquired by the task equipment. Task image features can be determined by extracting image features from the task images.

[0045] According to embodiments of this disclosure, the baseline task features of a target modality among multiple preset task features include baseline sub-features associated with preset behavioral events. The preset sub-features of each of the multiple preset task features are associated based on time attributes or event attributes. The baseline task features may include preset task features corresponding to task behavior data of the target modality. The baseline sub-features in the baseline task features may be task behavior sub-data related to preset behavioral events within the task behavior data.

[0046] For example, the work behavior data represents the work audio data collected from 10:00 to 11:00, and the baseline work features are determined by encoding the work audio data. The work audio sub-data corresponding to 10:10 to 10:20 represents the abnormal noise event of the transmission of the work vehicle. Therefore, the preset sub-feature corresponding to 10:10 to 10:20 in the baseline work features can be used as the target sub-feature corresponding to the abnormal noise event of the transmission.

[0047] In some embodiments, multiple benchmark sub-features corresponding to multiple modalities are associated with each other based on their respective acquisition timestamps as time attributes. Alternatively, multiple benchmark sub-features may also be labeled with event attribute fields such as event type and event development stage, so that multiple benchmark sub-features corresponding to multiple modalities can be associated based on event attribute fields.

[0048] According to embodiments of this disclosure, the target job feature includes target sub-features that match the target behavioral event. For example, the target job feature may include multiple target sub-features corresponding to multiple modalities, and the multiple target sub-features may be associated with the same event attribute field, or the multiple target sub-features may be associated with the same time attribute.

[0049] In one embodiment, a target audio sub-feature can be determined from multiple audio sub-features based on the event attribute field of the audio sub-feature annotation of the audio modality as the target audio sub-feature matching the gearbox abnormal noise event. Based on the time attribute field, preset sub-features corresponding to other modalities associated with the target audio sub-feature are determined as other target sub-features. Thus, image sub-features of image modalities and point cloud sub-features of point cloud modalities collected within the same or similar time periods can be retrieved as target sub-features based on the target sub-features. This allows the target object to query target sub-features of multiple modalities related to the target behavior event based on the input information, and then use the work behavior sub-data corresponding to the multiple target sub-features as multimodal target behavior data. This enables the target object to quickly and accurately retrieve multiple modal work behavior data matching the behavioral requirements from a large-scale stored work behavior data, improving the retrieval efficiency and accuracy of work behavior data.

[0050] In some embodiments, target job behavior data can be displayed on an interactive page where dialogue occurs with the target object, allowing the target object to quickly obtain the required data through dialogue. However, this is not limited to this; the target job behavior data can also be pushed to the target object by sending it to a storage area capable of receiving data, such as an email address specified by the target object. The embodiments of this disclosure do not limit the specific method of pushing target job behavior data to the target object.

[0051] According to embodiments of this disclosure, by identifying the behavioral event intent represented by the input information of the target object, benchmark sub-features matching the behavioral event intent are determined from benchmark operational features characterizing operational behavior. Furthermore, based on the time or event attributes used to associate multiple preset sub-features, other associated target sub-features are determined using the benchmark sub-features matching the behavioral event intent. This enables the accurate retrieval of multimodal target sub-features as target operational features according to the target object's behavioral event intent. Subsequently, by pushing target operational behavior data determined based on these target operational features to the target object, the data retrieval needs of the target object are met, improving data retrieval efficiency and accuracy. This, in turn, improves the efficiency of data retrieval for operational environments such as mining operations, and enhances the execution efficiency of specific tasks such as mining operation algorithm updates and terrain data updates.

[0052] According to embodiments of this disclosure, a plurality of preset job features include job audio features characterizing job behavior data of an audio modality, and specified job features characterizing job behavior data of a specified modality. The specified modality includes at least one of point cloud data modality and image modality, and the job audio features serve as a baseline job feature.

[0053] According to embodiments of this disclosure, the target modality can be an audio modality, and the operational audio features can be reference operational features. The operational audio features can be determined by encoding operational audio data; for example, the operational audio data can be Mel-spectrum data representing overtaking behavior, and the operational audio features can be obtained by encoding the Mel-spectrum data. The operational audio features can be aligned with multiple other specified operational features based on temporal attributes; for example, the operational audio features can be temporally aligned with operational image features representing multiple frames of operational images. The aligned multiple preset operational features can represent multiple modal operational behavior data collected by the operational equipment during the execution of the same operational behavior.

[0054] In some embodiments, the operational audio features include multiple audio sub-features corresponding to multiple preset behavioral events in the operational behavior. The multiple audio sub-features may represent multiple operational audio sub-data ordered based on time attributes, and the multiple operational audio sub-data may, for example, represent sequentially executed overtaking behavior, uphill behavior, reversing behavior, and unloading material behavior.

[0055] In some embodiments, by configuring behavioral attribute identifiers representing job behavior events for multiple audio sub-features, a target audio sub-feature matching the target behavior event can be determined from multiple audio sub-features based on the intent of the behavior event. This allows for the determination of point cloud sub-features, image sub-features, etc., aligned with the target audio sub-feature from multiple preset job features of other specified modalities, thereby determining target sub-features of multiple modalities related to the target behavior event and obtaining multimodal target job features.

[0056] In some embodiments, multiple audio sub-features are not associated with behavioral attribute identifiers. A trained large model or other deep learning model can be used to perform intent understanding on multiple audio sub-features based on the target behavioral event represented by the behavioral event intent, so as to determine the target audio sub-feature from multiple audio sub-features.

[0057] Since the audio data of the operation can be collected by audio acquisition devices set near the engine, transmission and other operating devices, the audio data generated during the operation of the operating device can be collected under conditions with less interference from the operating environment. This allows the sub-features of the operation to more accurately represent the complete process of the operating device's operation, avoids the disturbance of operating environment conditions such as fog, rain and snow on the collected operation behavior data, improves the matching degree between the sub-features of the operation and the target behavior event represented by the behavioral event, and further determines other modal target sub-features related to the target operation feature based on the target audio sub-features. This enables more accurate retrieval of target operation features of any modality related to the target behavior event based on dialogue, thereby improving the retrieval efficiency and accuracy of operation behavior data.

[0058] In some embodiments, multiple specified sub-features in a specified job feature are aligned with multiple preset behavioral events and multiple audio sub-features, wherein the specified sub-features represent the data content of a first time period in the job behavior data of a specified modality, and the audio sub-features represent the data content of a second time period in the job behavior data, and the first time period is shorter than the second time period.

[0059] Figure 3 A schematic diagram illustrating the preset operation features according to an embodiment of the present disclosure is shown.

[0060] like Figure 3 As shown, the preset task features can include task audio features in the audio modality, task image features in the image modality, and task point cloud features in the point cloud modality. The task audio features include multiple audio sub-features s101, s102, s103, and s104 arranged according to time attributes. Each of the multiple audio sub-features can correspond to an audio segment with a duration of 10 seconds. The data content of the second time period is an audio segment with a duration of 10 seconds.

[0061] The task image features include multiple image sub-features p101, p102, p103, and p104. These image sub-features can represent a 6-second task video segment, where the first time interval from time t0 to time t1' can represent a 6-second segment. The data content of the first time interval can be a 6-second task video segment. The multiple image sub-features p101, p102, p103, and p104, along with the multiple audio sub-features s101, s102, s103, and s104, are time-aligned based on times t0, t1, t2, and t3.

[0062] This allows for the reduction of data storage on storage devices by storing time-aligned job audio and image features. By using job audio features as baseline job features, a large model is employed for semantic understanding based on the periodic fluctuations of audio sub-features to determine the audio sub-feature s102 that matches the target behavioral event from multiple audio sub-features, serving as the target baseline audio sub-feature. Thus, the audio sub-feature s102 and image sub-feature p102 are used as target job features matching the retrieval intent of the input information. Video clips corresponding to image sub-feature p102 are pushed to the target object as target job behavior data, thereby achieving cross-modal job data retrieval according to the target object's job data retrieval needs and improving retrieval efficiency.

[0063] In some embodiments, the baseline job feature is a job audio feature characterizing job audio data. The job audio sub-features in the job audio feature represent job audio segments corresponding to preset behavioral events in the job behavior. Job audio segments can be understood as job audio sub-data. For example, job audio segments can be obtained by collecting audio data during the execution of preset behavioral events on the job equipment.

[0064] In some embodiments, determining a target task feature from a plurality of preset task features characterizing a task behavior based on the intent of a behavioral event may include: determining a target baseline audio sub-feature corresponding to the target behavioral event from the task audio features of the audio modality based on the intent of the behavioral event; and determining a target associated sub-feature associated with the target baseline audio sub-feature from a plurality of preset task features.

[0065] According to embodiments of this disclosure, the target sub-feature includes at least one of a target reference audio sub-feature and a target associated sub-feature. For example, a target point cloud sub-feature and a target job image sub-feature associated with the target reference audio sub-feature. Multiple target sub-features may include the target reference audio sub-feature and the target job image sub-feature, or multiple target sub-features may include the target reference audio sub-feature and the target point cloud sub-feature.

[0066] In some embodiments, a trained large model can be used to understand the intent of multiple task audio sub-features in the task audio features based on the intent of the behavioral event, thereby determining the target benchmark audio sub-features. The trained large model can be supervisedly fine-tuned based on sample audio features and the task behavioral event labels corresponding to the sample audio features to obtain target benchmark audio sub-features that can identify those that match the intent of the behavioral event.

[0067] In some embodiments, target association sub-features can represent image, video frame, point cloud, or structured signal (alarm, etc.) feature information that is time-aligned with the target reference sub-features of the audio modality. This allows for precise matching of the target object's input information representing the intent to retrieve operational behavior data based on the target reference audio sub-features that have a high degree of matching with the target behavioral event. Furthermore, it enables the determination of other modal target association sub-features associated with the target operational features based on the target audio sub-features, reducing errors caused by weather conditions, data acquisition device defects, etc. This allows for more accurate retrieval of target operational features of any modality related to the target behavioral event based on a dialogue-based approach, improving the retrieval efficiency and accuracy of operational behavior data.

[0068] In some embodiments, determining target baseline audio sub-features corresponding to the target behavioral event from the job features of the audio modality based on the behavioral event intent includes: obtaining reference audio spectral features that match the target behavioral event represented by the behavioral event intent, and determining target baseline audio sub-features that match the reference audio spectral features from the baseline job features;

[0069] According to embodiments of this disclosure, the reference audio spectrum features can be determined by feature extraction of audio data, and the reference audio spectrum features characterize audio data generated by at least one of the following behavioral events: uphill driving event, overtaking event, and unloading event.

[0070] For example, reference audio spectrum features can be used to encode and determine the operational audio data collected during the uphill driving behavior of a work vehicle. The operational audio data related to uphill driving can be obtained by collecting audio data from the engine and transmission of the work vehicle. In one example, the audio data represented by the reference audio spectrum features may include engine operating audio data and transmission operating audio data generated during the uphill driving behavior. Therefore, the engine operating audio data and transmission operating audio data can be aligned and encoded according to time attributes to obtain the reference audio spectrum features corresponding to the uphill driving event.

[0071] For example, the reference audio spectrum characteristics can be used to encode and determine the operational audio data collected during the overtaking maneuver of the work vehicle. The operational audio data related to the overtaking maneuver can be obtained by collecting audio data from the engine, transmission, and horn of the work vehicle.

[0072] In one example, the audio data represented by the reference audio spectrum features may include engine operating audio data, transmission operating audio data, and overtaking horn warning audio data generated during the overtaking maneuver by the engine, transmission, and horn. Because the transmission downshifts and the engine speed changes during the overtaking maneuver, and the vehicle sends overtaking horn warning audio data to other equipment to indicate the impending overtaking maneuver, the engine operating audio data, transmission operating audio data, and overtaking horn warning audio data can be aligned and encoded according to their time attributes to obtain reference audio spectrum features that accurately represent the overtaking event. This allows for more precise matching of the target operational audio sub-features corresponding to the overtaking event based on the reference audio spectrum features.

[0073] In one example, the audio data represented by the reference audio spectrum features may include audio data of the truck bed lifting and unloading horn prompts generated during the unloading event performed by the truck bed device and horn device. Furthermore, the audio data represented by the reference audio spectrum features may also include material unloading audio data, which represents the ambient sound data generated when materials are unloaded from the work vehicle onto the ground or other areas. Therefore, the multi-source audio generated by the work vehicle during the unloading event can be represented based on the reference audio spectrum features, enabling the reference audio spectrum features to accurately characterize the unloading events and other operational events performed by the work equipment.

[0074] By aligning and encoding the audio data of the truck bed lifting, the audio data of the unloading horn, and the audio data of the material unloading according to the time attribute, a reference audio spectrum feature that can accurately represent the unloading event is obtained. This allows for more precise matching of the target operation audio sub-features corresponding to the unloading event based on the reference audio spectrum feature, thereby improving the retrieval requirements of the target operation features and the behavioral event that represents the unloading event.

[0075] In some embodiments, target prompt words are determined based on reference audio spectral features corresponding to the target behavior event. A large model is then used to process multiple baseline audio sub-features in the baseline task features based on the target prompt words to obtain target baseline audio sub-features that match the target behavior event. This allows the large model to understand the reference audio spectral features based on the target prompt words, improving the matching degree between the target baseline audio sub-features and the target behavior event, thereby enhancing the retrieval efficiency of task behavior data related to the target behavior event through a dialogue-based approach.

[0076] In some embodiments, the baseline job feature is a job audio feature that characterizes job audio data, and the job audio sub-feature in the job audio feature represents a job audio segment corresponding to a preset behavior event type.

[0077] According to embodiments of this disclosure, the behavioral event intent includes a first intent characterizing the event type of the target behavioral event and a second intent describing the target event elements associated with the target behavioral event.

[0078] In some embodiments, based on a first intent, candidate baseline audio sub-features corresponding to the event type of the target behavior event are determined from the operational audio features. For example, a large model can be used to process the operational audio features according to the first intent, thereby enabling semantic understanding of the operational audio features based on the event type of the target behavior event represented by the first intent, and determining candidate audio sub-features matching the target behavior event type from the operational audio features. The candidate audio sub-features can represent audio data matching any event type, such as overtaking events, uphill driving events, unloading events, etc.

[0079] In some embodiments, based on a second intent, a large model is used to perform event element understanding on at least one of the candidate baseline audio sub-features and candidate related sub-features to obtain the target sub-feature.

[0080] According to embodiments of this disclosure, candidate association sub-features can be preset sub-features representing operational behavior data other than the audio modality. For example, candidate association sub-features can be operational point cloud sub-features representing operational point cloud data. Or, for another example, candidate association sub-features can be operational image sub-features representing operational images. Candidate association sub-features are associated with candidate reference audio sub-features based on time attributes, and can be determined from preset sub-features corresponding to other modalities based on the timestamp corresponding to the candidate reference audio sub-features.

[0081] In some embodiments, target event elements can represent the target object's input information requirements for event elements such as spatial elements, environmental elements, and anomaly type elements of the target behavior event. Event elements of a work behavior event can be understood as the event states generated during the execution of the work behavior event by the work equipment.

[0082] In some examples, spatial features can represent the location type, geographic coordinates, etc., of a work activity event. Spatial features can be, for example, unloading areas, uphill areas, etc., which represent the spatial characteristics of a work activity event.

[0083] In some examples, environmental elements can represent factors related to the work environment in the work behavior events performed by the working equipment. For example, environmental elements may include weather elements (such as snow, fog, and angle of light), time elements (such as night and day indicating the work period), and work object elements (such as elements that match the behavior events performed in combination with other working equipment such as mining trucks and excavators). Among these, work object elements can represent influencing factors such as the type and quantity of work objects involved in the execution of the target behavior event.

[0084] In some examples, the exception type element can represent factors that cause operational failures, such as the fault type element (e.g., the unloading device not starting, or abnormal stopping during uphill driving).

[0085] According to embodiments of this disclosure, target sub-features characterize data content related to target event elements in target operational behavior data. Target event elements can be determined using a large model based on semantic understanding of input information, or they can be determined using a large model based on intent understanding of dialogue content during a dialogue with a target object.

[0086] In some embodiments, a large model can be used to understand candidate related sub-features and candidate baseline audio sub-features based on the semantic meaning of text prompts for target event elements, thereby enabling the selection of target sub-features that match the semantic meaning of text prompts from preset sub-features of different modalities. This achieves cross-modal filtering of preset sub-features across multiple modalities based on target event elements, improving the retrieval efficiency and accuracy of target task features, meeting the diverse retrieval needs of target objects for task behavior data, and enhancing the retrieval efficiency of multimodal task behavior data.

[0087] In some embodiments, the large model can understand the behavioral event types represented by candidate baseline audio sub-features and multiple candidate related sub-features based on the semantic textual prompts of the target event elements. It can also sort the event types of the candidate behavioral events corresponding to the multiple candidate sub-features according to the preset time attributes of each sub-feature to determine the execution order of multiple work actions performed by the work equipment. This determines whether the work behavior data generated by the work equipment performing multiple work actions matches the target abnormal element that the target object needs to retrieve. Therefore, the large model can be used to retrieve target work behavior data that indicates abnormal behavior of the target equipment, defects in the work behavior sequence, etc., with relatively high accuracy, thereby improving the retrieval efficiency of abnormal target work behavior data for the target object.

[0088] According to embodiments of this disclosure, without event type labeling of preset sub-features, candidate baseline audio sub-features corresponding to the target behavior event type can be determined by using a large model to understand audio sub-features. This allows for the retrieval of multiple modalities of operational behavior data related to the event type of the target behavior event based on candidate associated sub-features of other modalities associated with the candidate baseline audio sub-features. By using a large model to understand the event elements of candidate baseline audio sub-features and candidate associated sub-features based on the target event elements, the cross-modal semantic understanding capability of the large model can be used to more accurately retrieve target operational features related to diverse and complex event elements without labeling preset operational features of multiple modalities, thereby improving the retrieval efficiency for target behavior data.

[0089] In some embodiments, event element understanding is performed using a large model on at least one of the candidate baseline audio sub-features and candidate correlation sub-features, including performing the following operations using the large model:

[0090] Semantic understanding is performed on multiple candidate event types corresponding to multiple candidate audio sub-features in the candidate job audio features to obtain abnormal event elements that represent the execution abnormality of the job behavior; based on the candidate job audio features corresponding to the abnormal event elements and other preset job features aligned with the time attributes of the candidate job audio features, multiple target job features are determined.

[0091] According to embodiments of this disclosure, candidate job audio features include candidate audio sub-features, and the candidate job audio features can be the job audio features where the candidate reference audio sub-features are located.

[0092] For example, the candidate assignment audio features are 10×1024 dimension feature vectors, and the candidate benchmark audio sub-features are 1×1024 dimension feature vectors in the first column of the candidate assignment audio features. The 1×1024 dimension feature vectors in columns 2 through 10 of the candidate assignment audio features represent nine different candidate audio sub-features. It should be understood that the candidate benchmark audio sub-features can also be candidate audio sub-features. Multiple candidate audio sub-features in the candidate assignment audio features can be arranged based on the temporal attributes of their respective audio segments.

[0093] According to embodiments of this disclosure, each of the multiple candidate audio sub-features in the candidate job audio features can represent an audio segment during the job execution process of the work equipment. The large model can determine the candidate event type corresponding to the candidate audio sub-feature by performing audio modal semantic understanding on the multiple candidate audio sub-features. The candidate event type can represent the behavior event type of the job behavior event represented by the audio segment corresponding to the candidate audio sub-feature. The multiple candidate event types are arranged in order based on the temporal attributes of the multiple candidate audio sub-features.

[0094] In one example, the first intent corresponding to the input information represents the demand intent for unloading events during the unloading operation of the work vehicle, and the second intent represents the target event element as an abnormal event element. By using a large model to determine candidate audio features related to the unloading event for event element understanding, the candidate event types corresponding to multiple candidate audio sub-features in the candidate audio features can be determined. Therefore, using preset abnormal events such as traffic jam events, engine stall events, and collision avoidance events corresponding to the abnormal event element, candidate audio sub-features corresponding to the abnormal event element can be determined from multiple candidate audio sub-features as target audio sub-features. By determining the target audio feature containing the target audio sub-feature as the target operation feature, and determining preset operation features of other modalities that are time-aligned with the target audio feature as other target operation features, it is possible to use the event types of the operation behavior events represented by the abnormal event element and the first intent to retrieve target operation features that match the complex demand intent of the target object relatively accurately through dialogue, even without labeling the preset operation features, thus meeting the diverse operation needs of the target object for model optimization, operation safety monitoring, etc.

[0095] In some embodiments, understanding the intent of the input information to obtain the intent of the behavioral event may further include: using a large model to perform semantic understanding on the input audio and video information to obtain input description information for describing the information content of the input audio and video information; and understanding the intent of the target object based on the input description information to obtain a first intent and a second intent.

[0096] According to embodiments of this disclosure, the input information includes input audio and video information. The input audio and video information may include any one or more of audio information, image information, and video information. A multimodal large model is used to encode the features of the input audio and video information, enabling the encoded input audio and video features to map input information from different modalities to the same feature space. The multimodal large model achieves semantic understanding of the multimodal input information by processing the input audio and video features.

[0097] In some embodiments, the input description information may be text content describing the video content of the input video information, or the input description information may further include descriptive text describing the job behavior event represented by the input audio information. The first intent characterizes the event type of the target behavior event, and the second intent describes the target event elements related to the target behavior event.

[0098] In one example, the input audio and video information includes video data representing a work vehicle driving uphill under snow-covered road conditions. The input description can be "driving uphill on the road in the image, with snow accumulation on the road." The large model performs semantic understanding on the input description information, outputting a first intent as "driving uphill" and a second intent as "snow-covered road." Here, "driving uphill" represents the event type of the target behavior event, and "snow-covered road" represents the target event element related to "driving uphill" in the second intent.

[0099] According to embodiments of this disclosure, the target event elements include at least one of the following: spatial elements, environmental elements, and work object elements. The spatial elements characterize the execution location of the behavioral event; the environmental elements are related to the work execution environment of the behavioral event; and the work object elements characterize the work object performing the work behavior.

[0100] By leveraging large models to perform semantic understanding of multimodal input audio and video information to determine the input description information, target objects can retrieve job behavior data by inputting input information of any modality. The input description information more accurately represents the target object's retrieval needs for the job behavior data to be retrieved. Therefore, multiple target job features can be determined based on the first and second intentions identified by the input description information, and multimodal target job behavior data corresponding to each target job feature can be fed back to the target object. This improves the retrieval efficiency of job behavior data under complex job conditions, reduces the retrieval process of the target object editing dialogue retrieval text, and enhances the optimization efficiency of job tasks in complex job scenarios.

[0101] In some embodiments, multiple preset job features are associated with multiple preset event elements. For example, preset job features can be labeled to associate them with corresponding preset event elements.

[0102] According to embodiments of this disclosure, determining a target job feature from a plurality of preset job features characterizing job behavior based on the intent of a behavioral event may include: determining candidate job features that match the target event element from a plurality of preset job features based on a second intent; and determining a target baseline job feature and associated job feature corresponding to the event type of the target behavioral event from the candidate job features based on a first intent.

[0103] According to embodiments of this disclosure, candidate job features that match the target event features can be determined from a plurality of preset job features based on the matching results between the target event feature and the target event feature represented by the second intent.

[0104] According to embodiments of this disclosure, associated job features are linked to target baseline job features based on time attributes; the target job features include target baseline job features and associated job features.

[0105] Therefore, by utilizing a large model to determine the target baseline operation features matching the target behavior event from candidate operation features based on the event type of the target behavior event represented by the first intent, and then determining related operation features as multiple target operation features based on the time or event attributes of the target baseline operation features from the candidate operation features, the following approach can be used: With pre-defined event elements labeled, the second intent is first used to select candidate operation features for a specific space or work environment based on the labeled event elements. Then, the target operation features are determined from the candidate operation features based on the event type corresponding to the first intent. This allows for rapid event type retrieval from candidate operation features matching the target event elements, thereby improving the retrieval efficiency of target operation behavior data.

[0106] In some embodiments, the above method for retrieving job behavior data further includes: using a large model to process target job features and input information to output anomaly description information.

[0107] According to embodiments of this disclosure, the anomaly description information describes the anomalous execution elements of the target operation behavior data. The anomalous execution elements indicate that the statistical results of the target behavior events in the target operation behavior data meet preset anomaly conditions. For example, the anomaly description information describes that the execution duration of the parking behavior event performed by the work vehicle exceeds a preset parking duration threshold.

[0108] In some embodiments, the anomaly description information describes abnormal operational behavior where the statistical results of behavioral events meet preset difference conditions. For example, the anomaly description information could be "the vehicle slipped three times during its uphill journey in the video," or "it stopped twice during its uphill journey." This allows for semantic understanding of the anomaly description information based on a large model to determine the anomaly event elements as "stopping events" and "at least two times." This enables more precise retrieval of target operational behavior data based on the anomaly event elements expressed in the target object's second intent. This target operational behavior data represents multimodal information indicating that the operational vehicle experienced at least two stopping events during its uphill journey. This allows the target object to retrieve operational behavior data of the operational vehicle under specified abnormal operational conditions by inputting multimodal audio and video information through dialogue, thereby improving the efficiency of control and optimization of the operational equipment and enhancing the operational efficiency and safety of the work environment.

[0109] Figure 4 The illustration shows an application scenario of a method for retrieving job behavior data according to an embodiment of the present disclosure.

[0110] like Figure 4 As shown, the target object can interact with the server through the client, and the method provided in this embodiment can be executed by the server. The server may include multiple functional modules such as an interaction module, a prompt word module, a large model, a retrieval tool, an annotation tool, a mining area corpus, and a preset operation feature library. The target object can input multimodal input information such as text, images, and audio through dialogue on the client. The server pushes target operation behavior data to the client by executing the method provided in this embodiment.

[0111] The interaction module is used to interact with the client. For example, the interaction module can be built on an agent module for dialogue. The agent module can perform task planning based on the context of the input information and the dialogue content, and call large models or other tool resources to generate response information containing target task data.

[0112] The prompt word module can construct prompt words for generating response information based on mining area domain knowledge in the mining area corpus. For example, the prompt word module fills the prompt word template according to the input information and context content transmitted by the interaction module, constructing prompt words for the large model to understand. The prompt word template can be stored in the mining area corpus, and the prompt word module can generate prompt words such as "Are there any vehicles waiting to be operated at the crushing station?", "Are there any available parking spaces in the spoil disposal area?", and "Are there any vehicles operating at the loading station?" based on the prompt word template and mining area domain knowledge. The mining area corpus can also structurally store core event element information such as spatial elements, behavioral elements, work object elements, and basic elements, realizing the digitization of mining area domain knowledge and facilitating the execution of retrieval methods by the large model.

[0113] The large model can vectorize multimodal work behavior data collected by equipment, obtaining vectorized representations of work behavior data of any modality, such as audio, image, and text data. This allows the preset work behavior features to contain higher-dimensional implicit feature information and store feature information with high density. Simultaneously, the large model can align preset work behavior features from multiple modalities according to temporal attributes to facilitate subsequent work behavior data retrieval. Multiple temporally aligned preset work features are stored in a preset work feature library. For example, the large model can use algorithms such as Cross-Attention, CLIP coding, and Transformer network layers to vectorize multimodal work behavior data, obtaining preset work features. Among the multiple temporally aligned preset work features in the preset work feature library, audio modality work audio features can be used as the baseline work feature. Preset work features from other modalities are temporally aligned with the work audio features.

[0114] The retrieval tool can be built based on any type of retrieval service tool, such as a retrieval engine architecture or a distributed retrieval service cluster. By receiving retrieval parameters transmitted from an interactive module or a large model, the retrieval tool can efficiently handle large-scale feature vector retrieval requests. The retrieval tool can include a retrieval algorithm module and a retrieval result processing module. The retrieval tool retrieves the target job features by performing vector retrieval in a pre-defined job feature library.

[0115] The interaction module determines the target job behavior data corresponding to the target job features from the database based on the target job features fed back by the retrieval tool, and pushes response information containing the target job behavior data to the client. The response information may also include the response text content output by the large model. The response text content can be used to describe the name of the target job behavior data, or to explain and describe the target job behavior data, so that the target object can engage in dialogue interaction based on the response information to complete further retrieval tasks.

[0116] The annotation tool can annotate the preset job features in the preset job feature library based on the event type and corresponding target event elements of the target behavior events identified by the large model, so that the preset job features can be retrieved and the retrieval task can be completed based on the labels of the preset job features.

[0117] In one example, the target object inputs the first input information: "Filter data from the first week of September in the xx mining area, filtering out scene data containing mining trucks and dust." The large model processes the first input information to execute the method for retrieving operational behavior data provided in this embodiment of the disclosure, and calls the retrieval tool to determine the target operational features corresponding to "xx" target operational behavior data from a preset operational feature library, thereby determining the target operational behavior data. The interaction module pushes the first response information 401 to the client to display multiple target operational images and response text describing the operational images to the target object.

[0118] In one example, the target object inputs a second input, which may include the text "Filter for rainy scenes similar to the images below," and images related to the operation of vehicles in rainy weather. The large model processes this second input to execute the method for retrieving operational behavior data provided in this embodiment of the disclosure, and invokes a retrieval tool to retrieve target operational features from a preset operational feature library. These target operational features represent rainy driving scene videos that serve as target operational behavior data. The interaction module pushes a second response message 402 to the client to display the rainy driving scene video and the response text used for the rainy driving scene video to the target object.

[0119] In one example, the target object inputs a third input, which may include the text "Filter out audio related to dumping construction waste, similar to the following sound," and example audio data related to the unloading operation performed by the work vehicle. The large model processes this third input to execute the method for retrieving work behavior data provided in this embodiment of the disclosure, and invokes a retrieval tool to retrieve target work features from a preset work feature library. The target work features represent the unloading operation audio data, which serves as the target work behavior data. The interaction module pushes a third response message 403 to the client to display the unloading operation audio data and a response text for a video of a rainy driving scenario to the target object.

[0120] Furthermore, when the target object confirms the use of the target task behavior data in the response information through the client, a labeling tool is used to annotate preset task features, associating these features with the target task behavior event and its elements. Additionally, the labeling tool can fine-tune the large model using the annotated preset task features, adjusting its parameters to more accurately identify the primary and secondary intents of the target object's input information, thereby optimizing the accuracy and efficiency of task behavior data retrieval. The server integrates model training and task behavior data retrieval into a single system to address the shortcomings of fragmented system functionality and improve the overall system functionality of the server.

[0121] Figure 5 A block diagram of an apparatus for retrieving job behavior data according to an embodiment of the present disclosure is shown schematically.

[0122] like Figure 5 As shown, the device 500 for retrieving job behavior data includes a receiving module 510, an intent understanding module 520, a determination module 530, and a push module 540.

[0123] The receiving module 510 is used to receive input information from the target object.

[0124] The intent understanding module 520 is used to understand the input information and obtain the behavioral event intent. The behavioral event intent represents the data retrieval of target behavioral events related to the work behavior performed by the work object.

[0125] The determination module 530 is used to determine the target operation feature from multiple preset operation features representing operation behavior based on the intent of the behavior event. The multiple preset operation features correspond to multiple operation behavior data of multiple modalities. The baseline operation features of the target modality among the multiple preset operation features include baseline sub-features associated with preset behavior events. The preset sub-features of each of the multiple preset operation features are associated based on time attributes or event attributes. The target operation feature includes target sub-features that match the target behavior event.

[0126] The push module 540 is used to push target operation behavior data determined based on the target operation characteristics to the target object.

[0127] According to an embodiment of this disclosure, the baseline operation feature is an operation audio feature that characterizes the operation audio data, and the operation audio sub-feature in the operation audio feature represents the operation audio segment corresponding to the preset behavior event in the operation behavior; wherein, the determination module includes: a first determination unit and a second determination unit.

[0128] The first determining unit is used to determine the target reference audio sub-features corresponding to the target behavioral event from the operational audio features of the audio modality based on the intent of the behavioral event.

[0129] The second determining unit is used to determine, from a plurality of preset operational features, a target associated sub-feature that is associated with a target reference audio sub-feature, wherein the target sub-feature includes at least one of the target reference audio sub-feature and the target associated sub-feature.

[0130] According to embodiments of this disclosure, the first determining unit includes: a first acquiring subunit and a first determining subunit.

[0131] The first acquisition subunit is used to acquire reference audio spectral features that match the target behavioral event that represents the intent of the behavioral event.

[0132] The first determining sub-unit is used to determine the target reference audio sub-features that match the reference audio spectrum features from the reference operation features.

[0133] According to embodiments of this disclosure, reference audio spectrum features characterize audio data generated by at least one of the following behavioral events: uphill driving event, overtaking event, and unloading event.

[0134] According to embodiments of this disclosure, the baseline task feature is a task audio feature characterizing task audio data. The task audio sub-features in the task audio feature represent: task audio segments corresponding to preset behavioral event types; behavioral event intents include a first intent characterizing the event type of the target behavioral event and a second intent describing the target event elements related to the target behavioral event; wherein, based on the first intent, candidate baseline audio sub-features corresponding to the event type of the target behavioral event are determined from the task audio features; based on the second intent, at least one of the candidate baseline audio sub-features and candidate related sub-features is used to understand the event elements using a large model to obtain the target sub-features, and the candidate related sub-features are associated with the candidate baseline audio sub-features based on time attributes, and the target sub-features characterize the data content related to the target event elements in the target task behavioral data.

[0135] According to embodiments of this disclosure, a large model is used to perform event element understanding on at least one of the candidate baseline audio sub-features and candidate related sub-features. This includes performing the following operations using the large model: performing semantic understanding on multiple candidate event types corresponding to multiple candidate audio sub-features in the candidate job audio features to obtain abnormal event elements that characterize the execution abnormality of the job behavior. The candidate job audio features include candidate baseline audio sub-features, and the multiple candidate event types are arranged in order based on the temporal attributes of the multiple candidate audio sub-features. Multiple target job features are determined based on the candidate job audio features corresponding to the abnormal event elements and other preset job features aligned with the temporal attributes of the candidate job audio features.

[0136] According to embodiments of this disclosure, the intent understanding module includes: a first obtaining unit and a second obtaining unit.

[0137] The first acquisition unit is used to perform semantic understanding on the input audio and video information using a large model to obtain input description information that describes the information content of the input audio and video information. The input information includes the input audio and video information.

[0138] The second obtaining unit is used to understand the intent of the target object based on the input description information to obtain a first intent and a second intent. The first intent represents the event type of the target behavior event, and the second intent describes the target event elements related to the target behavior event.

[0139] According to embodiments of this disclosure, the target event element includes at least one of the following: a spatial element, representing the execution location of the behavioral event; an environmental element, related to the work execution environment of the behavioral event; and a work object element, representing the work object that performs the work behavior.

[0140] According to embodiments of this disclosure, multiple preset operation features are associated with multiple preset event elements; the determining module includes a third determining unit and a fourth determining unit.

[0141] The third determining unit is used to determine, based on the second intent, candidate operation features that match the target event elements from multiple preset operation features.

[0142] The fourth determining unit is used to determine, based on the first intent, the target baseline operation feature and the associated operation feature corresponding to the event type of the target behavior event from the candidate operation features. The associated operation feature is associated with the target baseline operation feature based on the time attribute. The target operation feature includes the target baseline operation feature and the associated operation feature.

[0143] According to embodiments of this disclosure, the apparatus for retrieving job behavior data further includes an output module.

[0144] The output module is used to process the target job features and input information using a large model, and output anomaly description information. The anomaly description information describes the abnormal execution elements of the target job behavior data.

[0145] According to embodiments of this disclosure, the plurality of preset job features include job audio features representing job behavior data of an audio modality, and specified job features representing job behavior data of a specified modality. The specified modality includes at least one of point cloud data modality and image modality. The job audio features serve as a baseline job feature. The job audio features include a plurality of audio sub-features corresponding to a plurality of preset behavior events in the job behavior. The plurality of specified sub-features in the specified job features are aligned with the plurality of preset behavior events and the plurality of audio sub-features. The specified sub-features represent the data content of a first time period in the job behavior data of the specified modality, and the audio sub-features represent the data content of a second time period in the job behavior data. The first time period is shorter than the second time period.

[0146] Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be implemented by dividing them into multiple modules. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as hardware circuitry, such as a Field-Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a System-on-Chip, a System-on-a-Substrate, a System-on-Package, an Application-Specific Integrated Circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0147] For example, any plurality of the receiving module 510, intent understanding module 520, determination module 530, and push module 540 may be combined into one module / unit / subunit, or any one of these modules / units / subunits may be split into multiple modules / units / subunits. Alternatively, at least part of the functionality of one or more of these modules / units / subunits may be combined with at least part of the functionality of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of this disclosure, at least one of the receiving module 510, intent understanding module 520, determination module 530, and push module 540 may be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the receiving module 510, the intent understanding module 520, the determining module 530, and the pushing module 540 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0148] It should be noted that the apparatus part for retrieving job behavior data in the embodiments of this disclosure corresponds to the method part for retrieving job behavior data in the embodiments of this disclosure. For a detailed description of the apparatus part for retrieving job behavior data, please refer to the method part for retrieving job behavior data, which will not be repeated here.

[0149] Figure 6 A block diagram of an electronic device suitable for implementing the method for retrieving job behavior data described above, according to an embodiment of the present disclosure, is shown schematically. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0150] like Figure 6As shown, an electronic device 600 according to an embodiment of this disclosure includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a ROM (Read-Only Memory) 602 or a program loaded from a storage portion 608 into a RAM (Random Access Memory) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this disclosure.

[0151] RAM 603 stores various programs and data required for the operation of electronic device 600. Processor 601, ROM 602, and RAM 603 are interconnected via bus 604. Processor 601 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 602 and / or RAM 603. It should be noted that the programs may also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0152] According to embodiments of this disclosure, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to a bus 604. The electronic device 600 may also include one or more of the following components connected to the input / output (I / O) interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 610 as needed so that computer programs read from it can be installed into the storage section 608 as needed.

[0153] According to embodiments of this disclosure, the method flow according to embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by processor 601, it performs the functions defined in the system of embodiments of this disclosure. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0154] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0155] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0156] For example, according to embodiments of this disclosure, a computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0157] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this disclosure. When the computer program product is run on an electronic device, the program code enables the electronic device to implement the method for retrieving job behavior data provided in the embodiments of this disclosure.

[0158] When the computer program is executed by the processor 601, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0159] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 609, and / or installed from the removable medium 611. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0160] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0161] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features described in the various embodiments of the present disclosure can be combined and / or combined in various ways, even if such combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0162] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A method for retrieving job behavior data, comprising: Receive input information from the target object; The input information is subjected to intent understanding to obtain behavioral event intent, which represents the data retrieval of target behavioral events related to the work behavior performed by the work object; Based on the intent of the behavioral event, a target task feature is determined from multiple preset task features that characterize the task behavior. The multiple preset task features correspond to multiple task behavior data of multiple modalities. The baseline task features of the target modality among the multiple preset task features include baseline sub-features associated with preset behavioral events. The preset sub-features of each of the multiple preset task features are associated based on time attributes or event attributes. The target task feature includes target sub-features that match the target behavioral event. Push target operation behavior data determined based on the target operation characteristics to the target object.

2. The method according to claim 1, wherein, The benchmark operation feature is an operation audio feature that characterizes the operation audio data. The operation audio sub-features in the operation audio feature represent the operation audio segments corresponding to the preset behavior events in the operation behavior. The step of determining the target task feature from multiple preset task features characterizing task behavior based on the intent of the behavioral event includes: Based on the intent of the behavioral event, target baseline audio sub-features corresponding to the target behavioral event are determined from the operational audio features of the audio modality; and From a plurality of the preset job features, a target associated sub-feature is determined that is associated with the target reference audio sub-feature, wherein the target sub-feature includes at least one of the target reference audio sub-feature and the target associated sub-feature.

3. The method according to claim 2, wherein, Based on the intent of the behavioral event, target baseline audio sub-features corresponding to the target behavioral event are determined from the operational audio features of the audio modality, including: Obtain reference audio spectral features that match the target behavioral event that represents the intent of the behavioral event. Determine target reference audio sub-features that match the reference audio spectral features from the reference operation features; Preferably, the reference audio spectrum features characterize audio data generated by at least one of the following behavioral events: uphill driving event, overtaking event, and unloading event.

4. The method according to claim 1 or 2, wherein, The baseline task feature is a task audio feature that characterizes task audio data. The task audio sub-features in the task audio feature represent: task audio segments corresponding to preset behavioral event types; the behavioral event intent includes a first intent that characterizes the event type of the target behavioral event, and a second intent that describes the target event elements related to the target behavioral event. Based on the first intent, candidate baseline audio sub-features corresponding to the event type of the target behavior event are determined from the operation audio features; Based on the second intent, an event element understanding is performed on at least one of the candidate baseline audio sub-features and candidate related sub-features using a large model to obtain a target sub-feature. The candidate related sub-feature is associated with the candidate baseline audio sub-feature based on time attributes. The target sub-feature represents the data content in the target operation behavior data that is related to the target event element.

5. The method according to claim 4, wherein, The step of using a large model to perform event element understanding on at least one of the candidate baseline audio sub-features and candidate correlation sub-features includes performing the following operations using the large model: Semantic understanding is performed on multiple candidate event types corresponding to multiple candidate audio sub-features in the candidate job audio features to obtain abnormal event elements that characterize the execution abnormality of the job behavior. The candidate job audio features include the candidate baseline audio sub-features, and the multiple candidate event types are arranged in order based on the time attributes of the multiple candidate audio sub-features. Based on the candidate job audio features corresponding to the abnormal event elements, and other preset job features aligned with the time attributes of the candidate job audio features, a plurality of target job features are determined.

6. The method according to claim 1, wherein, The process of understanding the intent of the input information to obtain the intent of the behavioral event includes: A large model is used to perform semantic understanding on the input audio and video information to obtain input description information that describes the information content of the input audio and video information; the input information includes the input audio and video information; and Based on the input description information, the target object is subjected to intent understanding to obtain a first intent and a second intent. The first intent represents the event type of the target behavior event, and the second intent describes the target event elements related to the target behavior event. Preferably, the target event element includes at least one of the following: Spatial elements represent the location where the behavioral event is executed; Environmental factors are related to the operational environment in which the behavioral event is performed; The task object element represents the task object that performs the task behavior.

7. The method according to claim 6, wherein, Multiple preset task features are associated with multiple preset event elements; The process of determining the target job feature from multiple preset job features characterizing job behavior based on the intent of the behavioral event includes: Based on the second intent, candidate task features that match the target event element are determined from a plurality of preset task features; Based on the first intent, target baseline job features and associated job features corresponding to the event type of the target behavior event are determined from the candidate job features. The associated job features are associated with the target baseline job features based on the time attribute. The target job features include the target baseline job features and the associated job features.

8. The method according to claim 1, wherein, Also includes: By using a large model to process the target job features and the input information, anomaly description information is output, which describes the abnormal execution elements of the target job behavior data.

9. The method according to claim 1 or 2, wherein, The plurality of preset job features include job audio features that characterize job behavior data in an audio modality, and specified job features that characterize job behavior data in a specified modality. The specified modality includes at least one of point cloud data modality and image modality. The job audio features serve as the baseline job features. The task audio features include multiple audio sub-features corresponding to multiple preset behavioral events in the task behavior. The specified sub-features in the specified task features are aligned with the multiple preset behavioral events and the multiple audio sub-features. The specified sub-features represent the data content of the first time period in the task behavior data of the specified modality, and the audio sub-features represent the data content of the second time period in the task behavior data. The first time period is shorter than the second time period.

10. An apparatus for retrieving job behavior data, comprising: The receiving module is used to receive input information from the target object; The intent understanding module is used to understand the input information to obtain the behavioral event intent, which represents the data retrieval of target behavioral events related to the work behavior performed by the work object; The determination module is used to determine a target task feature from multiple preset task features characterizing task behavior based on the intent of the behavioral event. The multiple preset task features correspond to multiple task behavior data of multiple modalities. The baseline task features of the target modality among the multiple preset task features include baseline sub-features associated with preset behavioral events. The preset sub-features of each of the multiple preset task features are associated based on time attributes or event attributes. The target task feature includes target sub-features that match the target behavioral event. The push module is used to push target operation behavior data determined based on the target operation characteristics to the target object.

Images