Action space determination method and device of intelligent agent, equipment and storage medium

Abstract

The invention discloses an action space determination method and device of an intelligent agent, equipment and a storage medium, and relates to the field of artificial intelligence and machine learning. The method comprises the steps of determining target high-resolution visual features of a target object according to a collected time sequence picture of the target object based on a high-resolution visual model; determining target global visual features of the target object according to the time sequence picture based on a global visual model; preprocessing the initial noise, the initial diffusion step length and the historical time sequence action sequence of the target agent to obtain a target noise variable, a target diffusion step length and a target time sequence action sequence; and based on a multi-modal diffusion model and a decision diffusion model, according to the target high-resolution visual feature, the target noise variable, the target global visual feature, the target diffusion step length and the target time sequence action sequence, determining a target action space of the target agent. According to the technical characteristics, the accuracy and the reliability of determining the action space can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, in particular to the field of artificial intelligence and machine learning technology, and specifically to a method, device, equipment and storage medium for determining the action space of an intelligent agent. Background Art

[0002] End-to-end perception and decision-making models are widely used in fields such as autonomous driving, robot navigation, and intelligent monitoring. These models usually include visual perception modules and action space generation modules, which achieve efficient perception of the environment and intelligent decision-making through tight integration.

[0003] For the visual perception module, the current mainstream method uses a simple ResNet (Residual Network) architecture to extract visual perception information. Although ResNet performs well in image classification tasks, it lacks spatial perception of images and cannot provide sufficiently detailed and rich visual information. For the integration of perception and decision-making, the visual perception module fails to effectively guide decision-making experts, resulting in insufficient perception ability of decision-making experts. This deficiency makes it impossible for the system to accurately perceive the current visual environment and thus make accurate decisions. Summary of the invention

[0004] The present application provides a method, apparatus, device and storage medium for determining the action space of an intelligent agent to improve the accuracy and reliability of determining the action space.

[0005] According to one aspect of the present application, a method for determining an action space of an intelligent agent is provided, the method comprising:

[0006] Based on the high-resolution visual model, determining the target high-resolution visual features of the target object according to the collected time-series images of the target object;

[0007] Based on the global visual model, determining the target global visual features of the target object according to the time-series images;

[0008] Preprocess the initial noise, initial diffusion step and historical time-series action sequence of the target agent respectively to obtain the target noise variable, target diffusion step and target time-series action sequence;

[0009] Based on the multimodal diffusion model and the decision diffusion model, the target action space of the target agent is determined according to the target high-resolution visual features, the target noise variables, the target global visual features, the target diffusion step and the target temporal action sequence.

[0010] According to another aspect of the present application, a device for determining an action space of an intelligent agent is provided, the device comprising:

[0011] A first feature determination module is used to determine a target high-resolution visual feature of the target object based on a high-resolution visual model and according to the collected time-series images of the target object;

[0012] A second feature determination module is used to determine the target global visual features of the target object according to the time-series image based on a global visual model;

[0013] A data preprocessing module is used to preprocess the initial noise, the initial diffusion step and the historical time-series action sequence of the target agent respectively to obtain the target noise variable, the target diffusion step and the target time-series action sequence;

[0014] The action space determination module is used to determine the target action space of the target intelligent agent based on the multimodal diffusion model and the decision diffusion model according to the target high-resolution visual features, the target noise variables, the target global visual features, the target diffusion step size and the target temporal action sequence.

[0015] According to another aspect of the present application, an electronic device is provided, the electronic device comprising:

[0016] one or more processors;

[0017] A memory for storing one or more programs;

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement any one of the methods for determining the action space of an intelligent agent provided in the embodiments of the present application.

[0019] According to another aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method for determining the action space of any intelligent agent provided in the embodiments of the present application is implemented.

[0020] According to another aspect of the present application, a computer program product is provided, including a computer program, which, when executed by a processor, implements any method for determining the action space of an intelligent agent provided in an embodiment of the present application.

[0021] This application determines the target high-resolution visual features of the target object based on the collected time-series pictures of the target object based on a high-resolution visual model; determines the target global visual features of the target object based on the time-series pictures based on a global visual model; pre-processes the initial noise, initial diffusion step and historical time-series action sequence of the target agent respectively to obtain the target noise variable, target diffusion step and target time-series action sequence; determines the target action space of the target agent based on the target high-resolution visual features, target noise variables, target global visual features, target diffusion step and target time-series action sequence based on a multimodal diffusion model and a decision diffusion model. The above technical features, by introducing a high-resolution visual model, can provide more accurate and detailed visual information, which helps to improve the accuracy and reliability of determining the action space. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a flow chart of a method for determining an action space of an intelligent agent provided in accordance with Embodiment 1 of the present application;

[0023] Figure 2a is a flow chart of a method for determining an action space of an intelligent agent provided in Embodiment 2 of the present application;

[0024] Figure 2b This is the execution process of the multimodal diffusion model provided in Example 2 of the present application;

[0025] Figure 2c This is the execution process of the decision diffusion model provided in Example 2 of the present application;

[0026] Figure 3 is a structural diagram of a device for determining an action space of an intelligent agent provided in Embodiment 3 of the present application;

[0027] Figure 4 It is a schematic diagram of the structure of an electronic device that implements the method for determining the action space of an intelligent agent in an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] In addition, it should be noted that the collection, storage, use, processing, transmission, provision and disclosure of relevant data such as time-series images and target high-resolution visual features involved in the technical solution of this application are in compliance with the relevant laws and regulations and do not violate public order and good morals.

[0031] Embodiment 1

[0032] Figure 1 This is a flow chart of a method for determining the action space of an intelligent agent according to the first embodiment of the present application. This embodiment is applicable to the case of predicting the action to be performed by the intelligent agent, and can be performed by an action space determination device of the intelligent agent. The action space determination device of the intelligent agent can be implemented in the form of hardware and / or software. The action space determination device of the intelligent agent can be configured in a computer device, such as a server. Figure 1 As shown, the method includes:

[0033] S110 , based on a high-resolution visual model, determining target high-resolution visual features of the target object according to the collected time-series images of the target object.

[0034] Among them, the high-resolution visual model is a computer vision model that uses high-resolution image data to identify and extract the visual features of an object. Time-series images refer to continuous images collected in chronological order; each frame of the image records the state of the target object at different time points; these images usually contain key information about the object's motion, position and other dynamic changes. The target high-resolution visual features refer to the features extracted from the time-series images of the target object through the high-resolution visual model, which can represent the visual information of the target object and are suitable for the diffusion model; these features usually contain detailed information such as the appearance, shape, color, texture, etc. of the target.

[0035] It should be noted that in order to compensate for the problem of insufficient resolution of the visual perception module, the present application adopts more than one high-resolution visual model; exemplarily, the high-resolution visual model may include at least one of a target detection model, a semantic segmentation model, and an image classification model.

[0036] Among them, the target detection model refers to a model used to identify objects in images or videos, assign a category label to each object, and determine its location in the image; it requires not only identifying what the object is, but also determining the location of the object, usually represented by a bounding box. The semantic segmentation model refers to a model that classifies each pixel in the image so that each pixel is assigned a semantic label; unlike target detection, semantic segmentation not only identifies objects, but also needs to accurately segment the contours of the objects, especially in high-resolution images with rich details. The image classification model refers to a model that assigns the entire image to a specific category label, and is usually used to identify the category of the main object in the image without caring about the specific location or details of the object; for high-resolution images, the image classification model can extract features from richer image details and make more accurate classification judgments.

[0037] Optionally, the collected time-series images of the target object are respectively input into a target detection model and a semantic segmentation model to obtain position visual features, pixel visual features and category visual features of the target object; based on the position visual features, pixel visual features and category visual features, the target high-resolution visual features of the target object are determined.

[0038] Among them, position visual features refer to the spatial position or spatial coordinates of an object in an image; these features can be obtained through the bounding box of the target detection model, usually expressed as the position of the object in the image; the position visual features can include the position features, category features, confidence features, etc. of the target object. Pixel visual features refer to the visual information of each pixel in the image, usually described by the color, texture, brightness and other features of the pixel; in the semantic segmentation task, pixel visual features are usually expressed as the category label of each pixel or other more detailed visual information (such as texture, color depth, etc.); the pixel visual features can include pixel-level category features, spatial layout features, etc. Category visual features refer to feature information related to the category of the object; each object output by the target detection model is usually assigned a category label, such as "person", "car", "dog", etc.; category visual features are usually obtained through the classification layer of the neural network, describing the type of the object; the category visual features can include category features, attribute features, etc.

[0039] Furthermore, determining the target high-resolution visual features of the target object based on the position visual features, pixel visual features and category visual features can be: performing feature fusion on the position visual features, pixel visual features and category visual features to obtain candidate high-resolution visual features of the target object; and performing full connection and position embedding on the candidate high-resolution visual features to obtain the target high-resolution visual features of the target object.

[0040] Among them, candidate high-resolution visual features refer to high-dimensional features obtained by fusing visual features such as position, pixels and categories; by fusing these different features, the model can describe the target object at a higher resolution, which is usually used to accurately extract and express the details of the object.

[0041] Exemplarily, in order to fuse various visual features, the position visual features, pixel visual features and category visual features are flattened to make the feature dimensions consistent, and are fused through cascading of channel dimensions to obtain candidate high-resolution visual features; in order to make the visual features adapt to the diffusion model, the candidate high-resolution visual features are fully connected and embedded with position information to obtain the target high-resolution visual features of the target object.

[0042] S120 , based on the global visual model and according to the time-series images, determine the target global visual features of the target object.

[0043] Among them, the global vision model refers to a model used to process global image features, which usually extracts information related to the background and environment of the target object by analyzing the entire image; compared with local features (such as object edges and textures), global features focus on describing the overall relationship between objects and other elements in the image, such as the position, relative size, and viewing angle of the object. The target global visual feature refers to the feature extracted from the time-series image through the global vision model, which describes the relative relationship between the target object and its environment and is suitable for the diffusion model; it may contain information such as the position, motion trajectory, and relative angle of the target in the image, which is used to characterize the spatial and temporal information of the target object.

[0044] Optionally, the time-series image is input into a global visual model to obtain candidate global visual features of the target object; the candidate global visual features are fully connected and positionally embedded to obtain target global visual features of the target object.

[0045] Among them, the candidate global visual features refer to the features extracted from the temporal images through the global visual model, which describe the relative relationship between the target object and its environment; it may contain information such as the position, motion trajectory, relative angle, etc. of the target in the image, which is used to characterize the spatial and temporal information of the target object.

[0046] S130, preprocessing the initial noise, the initial diffusion step and the historical time-series action sequence of the target intelligent agent respectively to obtain the target noise variable, the target diffusion step and the target time-series action sequence.

[0047] Among them, the initial noise refers to the pre-randomly generated Gaussian noise with a mean of 0. The initial diffusion step size refers to the pre-set step size of each stage from the initial noise to the final generated target in the diffusion model. Target agent is an artificial intelligence system or agent whose core function is to achieve a specific goal through interaction with the environment; this goal is usually driven by the behavior of the agent, and the agent needs to achieve these goals through perception (obtaining environmental information), reasoning (making decisions), and execution (taking actions); for example, the agent can be a robot, a robotic arm, etc. The historical time series action sequence refers to the set of actions performed by the target agent in the past period of time; this sequence can provide background information for the current decision, help the agent understand its own behavior pattern in the environment, and make more accurate predictions and decisions. The target noise variable refers to the noise parameter introduced in the model for the action prediction process of the target agent. The target diffusion step size refers to the step size applied to the decision-making process of the target agent in the diffusion model; it determines how the diffusion model updates the action or state of the target agent by adjusting the noise information at each stage. The target temporal action sequence refers to the action sequence generated by the target agent based on its past behavior trajectory when performing a task; this sequence can help the model learn and predict the future actions of the target agent so as to adjust the strategy during task execution.

[0048] Specifically, the initial noise is fully connected and positionally embedded to obtain the target noise variable; the initial diffusion step is fully connected and positionally embedded to obtain the target diffusion step; the historical temporal action sequence of the target intelligent agent is fully connected and positionally embedded to obtain the target temporal action sequence.

[0049] S140. Based on the multimodal diffusion model and the decision diffusion model, the target action space of the target agent is determined according to the target high-resolution visual features, the target noise variables, the target global visual features, the target diffusion step size and the target temporal action sequence.

[0050] Among them, the multimodal diffusion model refers to a diffusion model that can handle multiple types of inputs (such as images, text, sound, etc.); it generates more comprehensive and rich outputs by fusing information from different modalities. The decision diffusion model refers to a model that uses the diffusion process to generate and optimize decisions; unlike traditional decision models, the decision diffusion model generates the best decision or action sequence by gradually adjusting noise and historical data; in reinforcement learning and decision systems, this type of model is used to deal with complex decision spaces and uncertain environments. The target action space refers to the set of all possible actions that the target agent can choose; according to the task requirements of the agent, the action space is usually limited to a set of specific action choices, and the target agent is to find the best action or decision in this space.

[0051] The embodiment of the present application determines the target high-resolution visual features of the target object based on the collected time-series pictures of the target object based on the high-resolution visual model; determines the target global visual features of the target object based on the time-series pictures based on the global visual model; pre-processes the initial noise, initial diffusion step and historical time-series action sequence of the target agent respectively to obtain the target noise variable, target diffusion step and target time-series action sequence; determines the target action space of the target agent based on the multimodal diffusion model and the decision diffusion model according to the target high-resolution visual features, target noise variables, target global visual features, target diffusion step and target time-series action sequence. The above technical features, by integrating professional models such as target detection model, semantic segmentation model and image classification model and performing multimodal feature fusion, enable the visual perception module to capture scene details from multiple aspects, thereby significantly enhancing the accuracy and richness of visual perception, and prompting the system to perceive more key visual information points. This is particularly critical for obstacle recognition or robot path planning in autonomous driving.

[0052] Embodiment 2

[0053] Figure 2a This is a flow chart of a method for determining the action space of an intelligent agent provided in Example 2 of the present application. Based on the technical solutions of the above-mentioned embodiments, this embodiment refines "based on a multimodal diffusion model and a decision diffusion model, according to the target high-resolution visual features, target noise variables, target global visual features, target diffusion step lengths and target temporal action sequences, the target action space of the target intelligent agent" into "feature integration of global visual features, target diffusion step lengths and target temporal action sequences to obtain conditional features; based on a multimodal diffusion model and a decision diffusion model, according to the target high-resolution visual features, target noise variables and conditional features, the target action space of the target intelligent agent". It should be noted that for the parts not described in detail in the embodiments of the present application, please refer to the relevant statements of other embodiments. Figure 2a As shown, the method includes:

[0054] S210 , based on a high-resolution visual model, determining target high-resolution visual features of the target object according to the collected time-series images of the target object.

[0055] S220 , based on the global visual model and according to the time series images, determine the target global visual features of the target object.

[0056] S230, preprocessing the initial noise, initial diffusion step and historical time-series action sequence of the target agent respectively to obtain a target noise variable, a target diffusion step and a target time-series action sequence.

[0057] S240, integrating the global visual features, the target diffusion step length, and the target temporal action sequence to obtain conditional features.

[0058] Among them, conditional features refer to the conditional information that the model relies on when making predictions or decisions; these features are obtained by combining multiple input features (such as global visual features, target diffusion steps, and target temporal action sequences) to describe the conditions of the model's current state. Feature integration refers to combining features from different sources or different levels to form a unified feature representation.

[0059] S250. Based on the multimodal diffusion model and the decision diffusion model, the target action space of the target agent is determined according to the target high-resolution visual features, target noise variables and conditional features.

[0060] Optionally, the target high-resolution visual features, target noise variables and conditional features are input into a multimodal diffusion model to obtain the candidate action space of the target agent; the candidate action space, conditional features and target noise variables are input into a decision diffusion model to obtain the target action space of the target agent.

[0061] Among them, the candidate action space refers to the set of all possible action selections generated based on input information (such as visual features of the target, noise variables, conditional features, etc.); in the multimodal diffusion model, the candidate action space represents a set of preliminary actions that the agent can take in a given state.

[0062] In an alternative embodiment, in combination Figure 2b and Figure 2c The execution flow of the multimodal diffusion model and the decision diffusion model shown in FIG. 1 ; the generation of the candidate action space and the target candidate action space can be achieved through the following steps:

[0063] 1) For the multimodal diffusion module, the conditional features are outputted as scaling coefficients, translation coefficients and gating coefficients after passing through the fully connected layer; the formula is as follows:

[0064] α,β,g=MLP(C);

[0065] Among them, α refers to the scaling factor. β refers to the translation factor. g refers to the gating factor. C refers to the conditional feature. MLP stands for Multilayer Perceptron, which is a feedforward neural network consisting of an input layer, a hidden layer, and an output layer.

[0066] 2) Scale and translate the target high-resolution visual features and target noise variables respectively; the formula is as follows:

[0067] M = α*M + β;

[0068] N = α * N + β;

[0069] Where M refers to the target high-resolution visual features and N refers to the target noise variable.

[0070] 3) In order to fully integrate the target high-resolution visual image features and the target noise variables, M and N are introduced into the multimodal attention operator; the formula is as follows:

[0071] K = Concat(M, N)*W k ;

[0072] Q = Concat(M, N)*W q ;

[0073] v = Concat(M, N)*W v ;

[0074]

[0075] Among them, K refers to the key matrix, which represents the "index" or "identity" in the self-attention mechanism; in the attention calculation, the key is used to compare with the query to calculate the attention weight. Q refers to the query matrix, which represents the object that the model needs to compare with other information; when calculating the attention score, the query will be matched with the key. V refers to the value matrix, which represents the actual information that needs to be weighted; after the attention score is calculated, the final output will be obtained through the weighted average. Concat(M, N) refers to the concatenation of visual features and noise variables into a unified representation. W k , W q and W v Refers to the weight matrix, which is used to map the features of the interface to the key, query and value space. Attenttion(K, Q, V) refers to the attention output. Softmax() refers to normalizing the similarity matrix to obtain the attention weight. QK T refers to the similarity between the query and the key. d refers to the dimension of the key vector. K T is the transpose of the key matrix.

[0076] 4) Enter the feedforward layer for processing; the formula is as follows:

[0077] M=FFN M (Attention);

[0078] N = FFN N (Attention);

[0079] Where M refers to the high-resolution visual features after being processed by the feed-forward layer. N refers to the noise variable after being processed by the feed-forward layer. FFN() refers to a feed-forward neural network, which is used to perform nonlinear transformations on the input.

[0080] 5) In each generation step, a gating layer is introduced to filter out noise and unwanted visual features; the formula is as follows:

[0081] M=M+g*M;

[0082] =N+g*N;

[0083] Here, g is the gating coefficient.

[0084] 6) Since the final action space is much smaller than the visual perception space, downsampling is used to gradually approach this action space. Since the features that need to be focused on have been strengthened through the gating layer, the feature points that need to be retained have larger values, so the maximum pooling is chosen for downsampling.

[0085] 7) Repeat the operation of the multimodal diffusion model multiple times until the candidate action space is output.

[0086] 8) Because the algorithm ultimately outputs the action space, in order to better adapt to high-dimensional action spaces and output stable action sequences, after multiple multimodal operations, the high-resolution visual expert features are discarded and the focus is on the generation of the action decision space. The decision diffusion module process is different except for the self-attention operator and the multimodal attention operator, while other process algorithms such as the conditional feature method remain the same; the formula for determining the self-attention operator is as follows:

[0087] K=N*W k ;

[0088] Q=N*W q ;

[0089] V=N*W v ;

[0090] Where N is the target noise variable.

[0091] 9) Repeat the decision diffusion model multiple times.

[0092] 10) The target action space is finally output, including joint positions, joint velocities, posture information, etc. The target agent will make relevant action decisions based on the target action space.

[0093] The embodiment of the present application determines the target high-resolution visual features of the target object based on the collected time-series pictures of the target object based on the high-resolution visual model; determines the target global visual features of the target object based on the time-series pictures based on the global visual model; pre-processes the initial noise, the initial diffusion step and the historical time-series action sequence of the target agent respectively to obtain the target noise variable, the target diffusion step and the target time-series action sequence; integrates the global visual features, the target diffusion step and the target time-series action sequence to obtain the conditional features; based on the multimodal diffusion model and the decision diffusion model, determines the target action space of the target agent according to the target high-resolution visual features, the target noise variables and the conditional features. The above technical features, by introducing the attention mechanism in the multimodal diffusion module, fully integrate the high-resolution visual features with the action generation noise, so that the generated action decision space is guided by the fine visual information, and the accuracy and stability of the action generation are improved. Secondly, the global features are introduced in the conditional input, so that the model can consider the overall environmental information when generating actions, and enhance the perception of the global context. In addition, by introducing the gating layer, the gating coefficient is used to modulate the features, emphasize important information, and filter noise and unnecessary features. By adopting a downsampling strategy and using the maximum pooling method to retain key information, the feature space is gradually approached to the action space, which better adapts to the generation requirements of action decisions and enhances the model's extraction and utilization of key features. At the same time, a separate diffusion process is designed for the action space, making the entire process more suitable for the generation of action decisions.

[0094] Embodiment 3

[0095] Figure 3 This is a schematic diagram of the structure of an intelligent agent action space determination device provided in the third embodiment of the present application, which can be applied to the case of predicting the action to be performed by the intelligent agent. The intelligent agent action space determination device can be implemented in the form of hardware and / or software. The intelligent agent action space determination device can be configured in a computer device, such as a server. Figure 3 As shown, the device comprises:

[0096] A first feature determination module 310 is used to determine a target high-resolution visual feature of a target object based on a high-resolution visual model and according to the collected time-series images of the target object;

[0097] A second feature determination module 320 is used to determine a target global visual feature of a target object according to the time-series image based on a global visual model;

[0098] A data preprocessing module 330 is used to preprocess the initial noise, the initial diffusion step and the historical time-series action sequence of the target agent respectively to obtain a target noise variable, a target diffusion step and a target time-series action sequence;

[0099] The action space determination module 340 is used to determine the target action space of the target agent based on the multimodal diffusion model and the decision diffusion model according to the target high-resolution visual features, target noise variables, target global visual features, target diffusion step size and target temporal action sequence.

[0100] The embodiment of the present application determines the target high-resolution visual features of the target object based on the collected time-series pictures of the target object based on the high-resolution visual model; determines the target global visual features of the target object based on the time-series pictures based on the global visual model; pre-processes the initial noise, initial diffusion step and historical time-series action sequence of the target agent respectively to obtain the target noise variable, target diffusion step and target time-series action sequence; determines the target action space of the target agent based on the target high-resolution visual features, target noise variables, target global visual features, target diffusion step and target time-series action sequence based on the multimodal diffusion model and the decision diffusion model. The above technical features can provide higher precision and detailed visual information by introducing a high-resolution visual model, which helps to improve the accuracy and reliability of determining the action space.

[0101] Optionally, the high-resolution visual model includes a target detection model, a semantic segmentation model, and an image classification model; accordingly, the first feature determination module 310 includes:

[0102] The model processing unit is used to input the collected time-series images of the target object into the target detection model and the semantic segmentation model respectively to obtain the position visual features, pixel visual features and category visual features of the target object;

[0103] The first feature determination unit is used to determine the target high-resolution visual features of the target object according to the position visual features, the pixel visual features and the category visual features.

[0104] Optionally, the first feature determination unit is specifically configured to:

[0105] Perform feature fusion on position visual features, pixel visual features and category visual features to obtain candidate high-resolution visual features of the target object;

[0106] The candidate high-resolution visual features are fully connected and positionally embedded to obtain the target high-resolution visual features of the target object.

[0107] Optionally, the second feature determination module 320 is specifically configured to:

[0108] Input the time-series images into the global visual model to obtain candidate global visual features of the target object;

[0109] The candidate global visual features are fully connected and positionally embedded to obtain the target global visual features of the target object.

[0110] Optionally, the action space determination module 340 includes:

[0111] The feature integration unit is used to integrate the global visual features, the target diffusion step and the target temporal action sequence to obtain the conditional features;

[0112] The action space determination unit is used to determine the target action space of the target agent based on the multimodal diffusion model and the decision diffusion model according to the target high-resolution visual features, target noise variables and conditional features.

[0113] Optionally, an action space determination unit is specifically used to:

[0114] Input the target high-resolution visual features, target noise variables and conditional features into the multimodal diffusion model to obtain the candidate action space of the target agent;

[0115] The candidate action space, conditional features and target noise variables are input into the decision diffusion model to obtain the target action space of the target agent.

[0116] The action space determination device of the intelligent agent provided in the embodiment of the present application can execute the action space determination method of the intelligent agent provided in any embodiment of the present application, and has the corresponding functional modules and beneficial effects for executing the action space determination method of each intelligent agent.

[0117] Embodiment 4

[0118] Figure 4 4 is a schematic diagram of the structure of an electronic device 410 that implements the action space determination method of the agent of the embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0119] like Figure 4As shown, the electronic device 410 includes at least one processor 411, and a memory connected to the at least one processor 411 in communication, such as a read-only memory (ROM) 412, a random access memory (RAM) 413, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 411 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 412 or the computer program loaded from the storage unit 418 to the random access memory (RAM) 413. In RAM413, various programs and data required for the operation of the electronic device 410 can also be stored. The processor 411, ROM412 and RAM413 are connected to each other via a bus 414. An input / output (I / O) interface 415 is also connected to the bus 414.

[0120] Multiple components in the electronic device 410 are connected to the I / O interface 415, including: an input unit 416, such as a keyboard, a mouse, etc.; an output unit 417, such as various types of displays, speakers, etc.; a storage unit 418, such as a disk, an optical disk, etc.; and a communication unit 419, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 419 allows the electronic device 410 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0121] The processor 411 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 411 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 411 executes the various methods and processes described above, such as the action space determination method of the intelligent agent.

[0122] In some embodiments, the action space determination method of the agent may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 418. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 410 via the ROM 412 and / or the communication unit 419. When the computer program is loaded into the RAM 413 and executed by the processor 411, one or more steps of the action space determination method of the agent described above may be performed. Alternatively, in other embodiments, the processor 411 may be configured as the action space determination method of the agent in any other appropriate manner (e.g., by means of firmware).

[0123] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0124] The computer programs for implementing the methods of the present application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable agent's action space determination device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0125] In the context of the present application, a computer readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device or equipment. A computer readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer readable storage medium may be a machine readable signal medium. A more specific example of a machine readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0126] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0127] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0128] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0129] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution of this application can be achieved, and this document is not limited here.

[0130] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A method for determining an action space of an intelligent agent, characterized in that: include: Based on the high-resolution visual model, determining the target high-resolution visual features of the target object according to the collected time-series images of the target object; Based on the global visual model, determining the target global visual features of the target object according to the time-series images; Preprocessing the initial noise, initial diffusion step and historical time-series action sequence of the target agent respectively to obtain the target noise variable, target diffusion step and target time-series action sequence; Based on the multimodal diffusion model and the decision diffusion model, the target action space of the target agent is determined according to the target high-resolution visual features, the target noise variables, the target global visual features, the target diffusion step and the target temporal action sequence.

2. The method according to claim 1, characterized in that: The high-resolution visual model includes a target detection model, a semantic segmentation model and an image classification model; accordingly, the high-resolution visual model is based on the collected time-series images of the target object to determine the target high-resolution visual features of the target object, including: The collected time-series images of the target object are input into the target detection model and the semantic segmentation model respectively to obtain the position visual features, pixel visual features and category visual features of the target object; A target high-resolution visual feature of the target object is determined according to the position visual feature, the pixel visual feature and the category visual feature.

3. The method according to claim 2, characterized in that The step of determining the target high-resolution visual feature of the target object according to the position visual feature, the pixel visual feature and the category visual feature comprises: Performing feature fusion on the position visual feature, the pixel visual feature and the category visual feature to obtain candidate high-resolution visual features of the target object; The candidate high-resolution visual features are fully connected and positionally embedded to obtain target high-resolution visual features of the target object.

4. The method according to claim 1, characterized in that: The determining, based on the global visual model and according to the time-series image, the target global visual features of the target object comprises: Inputting the time-series images into a global visual model to obtain candidate global visual features of the target object; The candidate global visual features are fully connected and positionally embedded to obtain target global visual features of the target object.

5. The method according to claim 1, characterized in that The method of determining the target action space of the target agent based on the multimodal diffusion model and the decision diffusion model according to the target high-resolution visual features, the target noise variables, the target global visual features, the target diffusion step length and the target temporal action sequence includes: Performing feature integration on the global visual feature, the target diffusion step and the target temporal action sequence to obtain a conditional feature; Based on a multimodal diffusion model and a decision diffusion model, a target action space of the target agent is determined according to the target high-resolution visual features, the target noise variables and the conditional features.

6. The method according to claim 5, characterized in that The method of determining the target action space of the target agent based on the multimodal diffusion model and the decision diffusion model according to the target high-resolution visual features, the target noise variables and the conditional features includes: Inputting the target high-resolution visual features, the target noise variables and the conditional features into a multimodal diffusion model to obtain a candidate action space of the target agent; The candidate action space, the conditional features and the target noise variable are input into a decision diffusion model to obtain the target action space of the target agent.

7. A device for determining an action space of an intelligent agent, characterized in that: include: A first feature determination module is used to determine a target high-resolution visual feature of the target object based on a high-resolution visual model and according to the collected time-series images of the target object; A second feature determination module is used to determine the target global visual features of the target object according to the time-series image based on a global visual model; A data preprocessing module is used to preprocess the initial noise, the initial diffusion step and the historical time-series action sequence of the target agent respectively to obtain the target noise variable, the target diffusion step and the target time-series action sequence; The action space determination module is used to determine the target action space of the target intelligent agent based on the multimodal diffusion model and the decision diffusion model according to the target high-resolution visual features, the target noise variables, the target global visual features, the target diffusion step size and the target temporal action sequence.

8. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method for determining the action space of an intelligent agent as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for determining the action space of an intelligent agent as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, which, when executed by a processor, implements the method for determining the action space of an intelligent agent according to any one of claims 1-6.

Images