A discrete action adaptive correction method and system for visual offline reinforcement learning, an electronic device, and a storage medium

Abstract

The application relates to the technical field of offline reinforcement learning, and provides a discrete action correction method and system for visual offline reinforcement learning and electronic equipment. The method comprises the following steps: acquiring an offline sample set, and generating a discrete latent action representation according to state transition related information in the offline sample set; generating an action evaluation result according to the discrete latent action representation, and determining a sample reliability result and an overall pollution degree result according to the relationship between the action evaluation result and original action labels in the offline sample set; adjusting the original action labels in the offline sample set according to the sample reliability result and the overall pollution degree result, so as to obtain updated action labels; and correcting the offline sample set according to the updated action labels. The embodiment of the application can weaken the influence of discrete action label pollution on subsequent training.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to technologies such as reinforcement learning, visual information processing, noise label correction, and offline decision optimization. Specifically, it relates to a discrete action correction method, system, electronic device, and storage medium for visual offline reinforcement learning. Background Technology

[0002] Reinforcement learning, a sequential decision-making technique in artificial intelligence, aims to enable an agent to learn actions and choose strategies based on feedback obtained through interaction with the environment, thereby improving long-term cumulative gains. Depending on whether continued interaction with the real environment is required during the training phase, reinforcement learning can generally be divided into online and offline reinforcement learning. Online reinforcement learning relies on continuous environmental interaction and trial and error during training. However, in high-cost, high-risk, or difficult-to-repeate applications such as robot control, autonomous driving, and industrial operations, online interaction is often limited by security, time costs, or equipment conditions. Therefore, it is usually necessary to use pre-collected and stored historical samples for training, i.e., offline reinforcement learning. For visual offline reinforcement learning tasks, the observations in the samples are usually image frames or video frame sequences. Related offline samples generally include observation data, action labels, reward information, and the next observation. Since the training process of offline reinforcement learning mainly relies on existing samples, the correctness of the correspondence between fields in the samples directly affects the effectiveness of subsequent value estimation and policy learning.

[0003] In visual offline reinforcement learning scenarios, the action space typically includes continuous action space and discrete action space. Continuous actions are usually represented by continuously adjustable numerical quantities. When there is a certain deviation in the action, the model may still be able to mitigate its impact through local adjustments in some cases. Discrete actions, on the other hand, usually correspond to a finite number of definite action categories, such as move, turn, grasp, release, or stop. Once the action label is recorded incorrectly, the action semantics will suffer from category-level mismatch, which is difficult to recover through continuous fine-tuning. Therefore, in visual offline reinforcement learning tasks, errors in discrete action labels are more likely to disrupt the correspondence between state changes and action labels in samples.

[0004] For a visual offline reinforcement learning sample, it can be represented as in, Represents observation data, Indicates action label, Indicates reward information, This indicates the observation at the next time step. When the action labels in a sample are contaminated, the corresponding sample can be represented as... in, This indicates a potentially erroneous action label. For discrete actions, if... If this happens, there will be an inconsistency between the state changes caused by the actual actions in the samples and the recorded action labels. As a result, the training process will no longer receive state-action samples that are consistent with the actual state transitions, but rather samples containing incorrect action semantics.

[0005] In existing technologies, common approaches to addressing noise or anomalies in offline reinforcement learning samples include: introducing robust loss, conservative estimation, anomaly sample weakening, or sample screening mechanisms during the value function training phase to reduce the impact of anomalies on value estimation and policy updates. Another common approach involves adding constraints, smoothing, or filtering to the training process to address observation noise, state perturbations, or out-of-distribution samples, thereby improving overall training stability. A common characteristic of these techniques is that their primary role is in the subsequent training phase; that is, suppressing the impact of anomalies during training, given that the samples have already been used as predetermined training inputs.

[0006] However, the aforementioned existing techniques have a major technical drawback in addressing the discrete action label contamination problem in visual offline reinforcement learning. Since these techniques do not directly alter the existing action labels in the samples, but primarily mitigate the influence of anomalous samples during value estimation or training constraint stages, when a discrete action label has already undergone a class-level error, the observed data, action label, and the next time-step observation in the sample still maintain an incorrect correspondence. For discrete actions, once the action label is incorrect, the action semantics carried by the sample have already undergone a discrete jump. Even if robust loss, conservative estimation, or anomalous sample suppression are used in subsequent training stages, they can usually only mitigate part of the impact of the erroneous sample during training, and cannot eliminate the inconsistency between the state change semantics in the sample and the action label itself. Therefore, existing techniques still cannot effectively solve the sample semantic mismatch problem caused by discrete action label contamination in visual offline reinforcement learning scenarios, thus affecting the stability and effectiveness of subsequent policy learning. Summary of the Invention

[0007] The present invention aims to provide a discrete action correction method, system, and electronic device for visual offline reinforcement learning, so as to at least solve the technical problem of inconsistency between the semantics of state changes and action labels in offline samples caused by discrete action label contamination in visual offline reinforcement learning scenarios.

[0008] To achieve the above objectives, embodiments of the present invention provide a discrete action correction method for visual offline reinforcement learning, comprising: acquiring an offline sample set, and generating a discrete latent action representation based on state transition information in the offline sample set; generating an action evaluation result based on the discrete latent action representation, and determining a sample reliability result and an overall contamination level result based on the relationship between the action evaluation result and the original action labels in the offline sample set; adjusting the original action labels in the offline sample set according to the sample reliability result and the overall contamination level result to obtain updated action labels; and correcting the offline sample set according to the updated action labels.

[0009] This invention also provides a discrete action correction system for offline reinforcement learning in vision, comprising: a latent action generation module for acquiring an offline sample set and generating discrete latent action representations based on state transition information in the offline sample set; an evaluation module for generating action evaluation results based on the discrete latent action representations and determining sample reliability results and overall contamination level results based on the relationship between the action evaluation results and the original action labels in the offline sample set; a label adjustment module for adjusting the original action labels in the offline sample set based on the sample reliability results and the overall contamination level results to obtain updated action labels; and a sample correction module for correcting the offline sample set based on the updated action labels.

[0010] This invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-described discrete motion correction method when executing the computer program.

[0011] This invention also provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-described discrete motion correction method.

[0012] Compared with the prior art, the embodiments of the present invention have at least the following beneficial effects: This invention generates discrete latent action representations based on state transition information in an offline sample set, generates action evaluation results based on the discrete latent action representations, and determines sample reliability and overall contamination levels based on the relationship between the action evaluation results and the original action labels. Therefore, it can determine the reliability of the original action labels without directly relying on the original action labels themselves, but based on the state change information in the offline samples.

[0013] In this embodiment of the invention, the original action labels are further adjusted based on the sample reliability results and the overall contamination level results to obtain updated action labels. The offline sample set is then corrected based on the updated action labels. Therefore, the sample semantic mismatch problem caused by discrete action label contamination can be reduced, and the consistency between state change information and action labels in offline samples can be improved, which is beneficial to improving the stability and effectiveness of subsequent policy learning. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be considered as limiting the scope of protection. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is the main flowchart of a discrete action correction method for visual offline reinforcement learning in one embodiment of the present invention; Figure 2 This is a diagram illustrating the generation process of discrete latent action representation in one embodiment of the present invention; Figure 3 This is a diagram illustrating the process of action evaluation, sample reliability analysis, and action label adjustment in one embodiment of the present invention; Figure 4 This is a flowchart illustrating the subsequent training based on the corrected offline sample set in one embodiment of the present invention; Figure 5 This is a key data flow relationship diagram in one embodiment of the present invention; Figure 6 This is a timing diagram of the collaborative execution of multiple processing entities in one embodiment of the present invention; Figure 7 This is a structural diagram of a discrete motion correction system according to one embodiment of the present invention; Figure 8 This is a structural diagram of an electronic device according to one embodiment of the present invention. Detailed Implementation

[0016] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention. Unless otherwise specified, the various embodiments and technical features in these embodiments can be combined with each other.

[0017] In one embodiment, the present invention addresses a technical problem where, when discrete action labels in offline visual reinforcement learning samples are contaminated, the correspondence between the observation data, action labels, and the observation at the next time step in the offline samples may no longer be consistent with the real state change. If subsequent training continues to directly rely on the original action labels, the state-action correspondence received during the training process will deviate from the action semantics implied by the real state transition in the samples, thereby affecting the stability and effectiveness of subsequent value estimation and policy learning. To address this technical problem, Figure 1 A main flowchart for discrete action correction in visual offline reinforcement learning is presented. This flowchart does not directly determine the correctness of the original action labels themselves. Instead, it first recovers the action semantics based on state transition information in the offline sample set. Then, it combines the relationship between the original action labels and the recovered action semantics to determine the sample reliability and overall contamination level. The original action labels are then adjusted, and the adjusted results are used to correct the offline sample set. Thus, the inconsistency between the original action labels and the state change semantics can be pre-processed before subsequent training.

[0018] In one embodiment, Figure 1 The process may include steps S110 to S140. Step S110 is used to obtain an offline sample set and generate a discrete latent action representation based on state transition information in the offline sample set; step S120 is used to generate an action evaluation result based on the discrete latent action representation, and determine the sample reliability result and the overall contamination level result based on the relationship between the action evaluation result and the original action labels in the offline sample set; step S130 is used to adjust the original action labels in the offline sample set based on the sample reliability result and the overall contamination level result to obtain updated action labels; step S140 is used to correct the offline sample set based on the updated action labels. Figure 1 The main process in the embodiment of the invention is expressed in an action-oriented manner to highlight the processing logic chain, rather than simply listing the participating components or modules. Figure 2 The discussion revolves around the generation process of discrete latent action representations. Figure 3 The discussion revolves around motion assessment, sample reliability analysis, and the process of adjusting motion labels. Figure 4 The additional processing chain for performing subsequent robust training based on the corrected offline sample set is further illustrated.

[0019] Step S110: Obtain an offline sample set and generate a discrete latent action representation based on the state transition information in the offline sample set.

[0020] exist Figure 1In the diagram, reference numeral 101 indicates the offline sample receiving stage, reference numeral 102 indicates the stage of extracting state transition-related information and modeling state changes, and reference numeral 103 indicates the stage of generating discrete latent action representations. The problem addressed in step S110 is not directly determining whether the original action label is correct, but rather establishing an action semantic recovery path relatively independent of the original action label. This is because when the original action label already contains discrete category-level errors, if the original action label itself is still used as the primary basis for action semantic recovery, the subsequent judgment process is easily influenced by the contaminated label, making it difficult to obtain action semantics consistent with the actual state changes. Therefore, in this step, state transition-related information from the offline sample set is preferentially used to generate discrete latent action representations, so that the recovery of action semantics is based on the state changes themselves.

[0021] In one embodiment, the offline sample set may consist of historically collected interaction trajectories, and the relevant samples may come from robot execution logs, navigation control records, operation trajectory files, game replay data, simulation environment cache data, or other offline storage media. For ease of explanation, the offline sample set received in step S110 can be represented as: (1) in, Represents the offline sample set. Indicates the number of samples. Indicates the first Observational data of one sample Indicates the first The original action labels of each sample, Indicates the first Reward information for each sample Indicates the first The next time-step observation of each sample. Since this offline sample set has already been collected and stored when it enters the processing link of this invention, therefore , , and All quantities can be considered known at the beginning of step S110. Subsequent processing in step S110 does not immediately affect... Instead of carrying out operations, priority should be given to... and The state change information carried between them is used to reconstruct the action semantics. In this way, the original action label... In this step, it is not used as the sole basis for generating action semantics, but is temporarily retained as an object for subsequent comparison and adjustment.

[0022] In one embodiment, the state transition information can directly include observation data and the next time-instance observation. That is, the step S110 of generating a discrete latent action representation based on the state transition information in the offline sample set can be expanded as follows: obtaining observation data and the next time-instance observation from the offline sample set as state transition information; constructing a state transition representation based on the observation data and the next time-instance observation; and generating a discrete latent action representation based on the state transition representation. Figure 2 This process is illustrated below. Figure 2 In the attached figures, reference numeral 201 indicates the observation input, reference numeral 202 indicates the observation input at the next time step, reference numeral 203 indicates the observation encoding stage, reference numeral 204 indicates the state transition representation generation stage, reference numeral 205 indicates the continuous latent action mapping stage, and reference numeral 206 indicates the discretization encoding stage. Thus, although step S110 is summarized at the main process level as acquiring an offline sample set and generating a discrete latent action representation based on the state transition information in the offline sample set, it already possesses a more feasible path at the specific implementation level.

[0023] In one embodiment, the observation input terminal 201 receives observation data from the sample. The next time-of-time observation input terminal 202 receives the next time-of-time observation from the sample. Since observation data and the next-time observation are typically represented as image frames, video frame sequences, or high-dimensional observation tensors obtained from visual sensors, they need to be transformed into intermediate features that facilitate state change analysis. The observation encoding stage indicated by reference numeral 203 in the attached figure can receive... and and output the corresponding observation features respectively. and subsequent observation features Their relationship can be expressed as: (2) (3) in, Represents the observation-encoding mapping, Indicates the data from observation Extracted observation features Indicates observation from the next moment. The extracted subsequent observation features. Because... and The quantity is already known in the sample set, and the observation-encoding mapping is... This can be achieved using a pre-trained or parameterized model employed in the current process, therefore and It can be calculated from known quantities, and thus become the known input in the subsequent state transition representation generation process.

[0024] In one embodiment, observation coding mapping This can be achieved using visual coding networks with local feature extraction capabilities, such as coding structures with convolutional feature extraction capabilities, attention-based visual feature extraction structures, spatiotemporal joint feature coding structures, or combinations of these structures. If the observed data... For a single-frame image, the observation-coding mapping... Spatial semantic features can be extracted directly from a single frame image; if the observed data For a multi-frame image sequence, the observation-coding mapping... Furthermore, dynamic information in the temporal dimension can be extracted based on spatial features. This is relevant for observations at the next moment. Alternatively, encoding paths with the same or shared parameters can be used for processing. The purpose of this processing is to transform the original high-dimensional pixel input into an intermediate representation that can be compared with subsequent state changes, so that the focus of subsequent steps shifts from the differences in the original pixels to the differences in the semantics of the encoded states.

[0025] In one different embodiment, observation coding mapping It is not required that the observed data and the next observation use exactly the same encoding parameters. If there are differences in sampling frequency, viewpoint variation, or modal enhancement between the signal sources of the observed data in the current vision task and the signal sources of the next observation, two different but functionally compatible encoding paths can be used to obtain the results. and As long as the subsequent state transition representation generation stage can jointly analyze these two types of features, information recovery based on state changes can be achieved. Therefore, the acquisition and encoding of state transition-related information is not limited to a single structure, but revolves around the technical essence of being able to extract comparable state features from observation data and the next moment's observation.

[0026] In observation features and subsequent observation features After being obtained, Figure 2 In the state transition representation generation step 204, a state transition representation is constructed based on these two known quantities. Their relationship can be expressed as: (4) in, This indicates that the state transition representation generates a mapping. Indicates sample The corresponding state transition representation. Because... and This has already been obtained from the aforementioned observation and coding process, therefore It can be further derived from known quantities. State transition representation. Its purpose is to no longer put and Instead of viewing them as two independent observations, we extract the common, different, or evolutionary relationships between them related to state changes. In other words, It is not equivalent to the static observation characteristics at a certain moment, but rather emphasizes the perspective from... arrive The state transition semantics implied in this change process.

[0027] In one embodiment, state transition representation generation mapping Features can be constructed using methods such as feature concatenation, feature differencing, feature interaction, or temporal fusion. For example, it can be used in the following form: (5) in, This indicates a splicing operation. Represents element-wise product. This represents the characteristic difference term, used to characterize the direction of state change. This represents the correspondence term, used to characterize the degree of coupling between features at corresponding positions at two time points. Because... and Since these are already known quantities, the difference terms, product terms, and concatenation results can all be further calculated to obtain the state transition representation. In this way, state transitions are represented. It can simultaneously preserve the differences, overlapping relationships, and coupling information between the states before and after an action, thereby providing richer information on state changes for subsequent semantic recovery of actions.

[0028] In one different embodiment, state transition representation Alternatively, a multi-level construction approach can be used. For example, the observation encoding stage 203 can output observation features and subsequent observation features at multiple semantic levels, while the state transition representation generation stage 204 extracts state change relationships at the shallow, middle, and high levels, respectively, and then merges the multi-level state change results into a unified state transition representation. The advantage of this approach is that different levels of features have varying sensitivities to local texture changes, geometric structure changes, and high-level semantic changes. Multi-level state transition representations can accommodate both fine-grained and coarse-grained state changes, thus more stably carrying the information required for action semantic recovery. Regardless of whether a single-layer or multi-layer approach is used, the technical essence lies in constructing state transition representations based on observation data and the next moment's observation, allowing state change information to be explicitly extracted first, and then further incorporated into the action semantic recovery process.

[0029] In state transition representation After being obtained, Figure 2 The continuous latent action mapping step 205 further generates a continuous latent action representation based on the state transition representation. Their relationship can be expressed as: (6) in, Represents latent action mapping, Indicates sample The corresponding continuous submerged actions are represented. Because... Since the quantity has already been obtained from the previous step, therefore It can also be derived further from known quantities. (This is related to the state transition representation.) In comparison, continuous submersible movements indicate The purpose of moving closer to the action semantic space is to further extract the potential information most relevant to the action factors from the state change representation, so that subsequent discretization processing can be carried out based on a more focused action semantic subspace.

[0030] In one embodiment, latent action mapping This can be achieved using a parameterized mapping structure with dimensionality compression and semantic aggregation capabilities, making... Redundant state changes unrelated to the action are suppressed, while action-related changes are retained. If... The dimension is denoted as Then we have: (7) in, express A continuous vector space. The values ​​can be set according to the scale of the action space, the complexity of the visual task, and the diversity of state changes. The specific values ​​do not constitute an unnecessary limitation on the scope of protection of this invention, as long as the dimension is sufficient to carry the potential semantics related to the action. Thus, state transition representation... After latent action mapping, it is no longer a state change feature in the general sense, but becomes a continuous latent action representation with action semantic orientation. .

[0031] In one different embodiment, latent action mapping Alternatively, a phased approach can be adopted. For example, the intermediate representation of the first potential action can be obtained first based on the state transition representation. And then according to Obtain the intermediate representation of the second potential action Ultimately by As a representation of continuous submersible movements : (8) (9) (10) in, and This represents the latent space mapping at different stages. Because... Since it is a known quantity, therefore , and All of these can be obtained sequentially. Through this multi-stage mapping method, coarse-grained action-related information contraction can be completed first, followed by fine-grained action semantic regularization, making the continuous latent action representation more compact and stable. Although the mathematical forms differ, its function remains the same: to progressively map state transition representations to a latent space more suitable for expressing action semantics.

[0032] Continuous submersible motion representation After being obtained, Figure 2 The discretization coding step 206 performs discretization processing to generate discrete latent action representations. Their relationship can be expressed as: (11) in, Represents a discretization mapping. Indicates sample The corresponding discrete latent action representation. Because... Since the quantities were already known in the previous stage, therefore It can also be calculated further. The purpose of discretization is to obtain the result from the still continuous potential space. This is further regularized to a finite number of discrete semantic units to facilitate the establishment of more stable action semantic boundaries between different samples. Especially in discrete action space scenarios, if pure continuous latent vectors are still used for direct evaluation, the statistical boundaries between samples may not be clear enough; while discrete latent action representation... It can compress action-related semantics into a more stable and easier-to-compare discrete representation, thereby creating conditions for the subsequent generation of action evaluation results and sample reliability analysis.

[0033] In one embodiment, discretization mapping This can be accomplished using a finite set of prototypes. Let the set of prototypes be: (12) in, Represents a set of prototypes. Indicates the first A discrete prototype This indicates the number of prototypes. The discretization coding step 206 can first be based on... The distance between each prototype is selected by choosing the index closest to the prototype. : (13) in, Represents the distance metric function. Indicates sample The corresponding prototype index. Because... and each prototype All of these can be considered known quantities at the current moment, therefore This can be further calculated. Then, based on the prototype index... Obtain the corresponding discrete latent action representation : (14) Thus, continuous latent motion representation It is normalized to a discrete semantic unit within a finite set of prototypes. Therefore, the specific implementation of generating discrete latent action representations based on state transition representations in this step has been formed from... arrive Then A complete closed loop.

[0034] In one different embodiment, discretization mapping Alternatively, a soft discrete approach can be used, where instead of outputting only a single prototype, a probability assignment vector is output for multiple prototypes, and then a discrete latent action representation is constructed based on this probability assignment vector. Let's assume that for... The probability of assigning each prototype is Then it can be expressed as: (15) (16) in, Indicates sample Belongs to the The probability of a prototype, This represents the discrete latent action representation obtained from soft assignment. Because... and Since it is a known quantity, therefore and All of these can be calculated. When using the soft discretization method, the discrete latent action representation not only retains the most important prototype attribution but also preserves the semantic information of potential neighboring actions. In scenarios where sample state changes are ambiguous or action switching boundaries are complex, the robustness of the discrete semantic representation can be improved. Even so, the technical essence of this implementation method remains unchanged, namely, generating discrete latent action representations based on state transition representations.

[0035] In one embodiment, step S110 has completed the following processing chain: first, receiving the offline sample set. Then, observation data are extracted from the sample. and the next moment observation As state transition related information, the observation features are then obtained through the observation coding step 203. and subsequent observation features The state transition representation is then obtained through the state transition representation generation step 204. Then, the continuous latent action representation is obtained through the latent action mapping step 205. Finally, the discrete latent action representation is obtained through the discretization encoding step 206. Thus, step S110 achieves the derivation from raw visual state change information to discrete action semantic representation. Compared to directly relying on raw action labels, this processing chain allows action semantic recovery to be based on evidence of state change. Since the raw action labels have not yet participated in the action semantic generation process, their contamination of the discrete latent action representation is negligible. The direct interference is weakened. This result provides a prerequisite for generating action evaluation results based on discrete latent action representations and analyzing the reliability of the original action labels in the subsequent step S120.

[0036] Step S120: Generate action evaluation results based on the discrete latent action representation, and determine the sample reliability results and overall contamination level results based on the relationship between the action evaluation results and the original action labels in the offline sample set.

[0037] exist Figure 1 In the attached diagram, reference numeral 104 indicates the action evaluation stage, and reference numeral 105 indicates the sample reliability and overall contamination level analysis stage. Step S120 follows the discrete potential action representation obtained in step S110. .because The action semantics during state transitions have already been extracted and normalized into the discrete latent space; therefore, further processing is now needed. Converted to be compatible with the original action tags The action evaluation results are compared, and the reliability of the current sample and the overall contamination level of the current sample set are judged based on the relationship between the two. In other words, step S110 is more focused on action semantic recovery, while step S120 is more focused on the consistency analysis between the action semantic recovery results and the original action labels.

[0038] In one embodiment, the step S120 of generating an action evaluation result based on the discrete latent action representation can be expanded to: generating an action prediction result based on the discrete latent action representation, and using the action prediction result as the action evaluation result. Figure 3 In the attached figure, reference numeral 301 indicates the motion evaluation generation step. The motion evaluation generation step 301 receives the discrete latent motion representation obtained in step S110. It also outputs the corresponding action prediction results. Their relationship can be expressed as: (17) in, Represents action evaluation mapping, Indicates sample The corresponding action prediction result. Because... This has already been obtained in step S110, therefore This can be further derived from known quantities. In this embodiment, As a result of motion evaluation, it serves as a bridge to convert discrete latent motion representations into representations that can be directly compared with the original motion labels.

[0039] In one embodiment, if the size of the discrete action space is Then the action prediction result It can be represented as a Dimensional vector: (18) in, Indicates sample Corresponding to the Predicted components for each action category. Action evaluation mapping. This can be achieved using classification mapping networks, prototype similarity estimation maps, or other structures capable of outputting action category scores based on discrete latent action representations. Because discrete latent action representations... The result is already obtained based on state transition information, therefore Instead of being directly derived from the original action labels, the action prediction distribution is obtained based on semantic recovery of state changes. Subsequently, it can be based on... Determine the predicted action label : (19) in, Indicates sample Predicted action labels. Due to It is known, therefore We can continue to calculate. At this point, the action prediction result is... and predict action labels All of these can be considered as known quantities in subsequent analysis, among which For more fine-grained consistency assessment, Used for subsequent labeling of the original action when conditions are met. Replace it.

[0040] In one different embodiment, the action evaluation result is not limited to a single prediction distribution, but can also be obtained jointly by multiple evaluation heads. Let there be a total... The first action evaluation head, then the second action evaluation head. The output of each evaluation head is denoted as Then it can be expressed as: (20) Subsequently, a comprehensive action evaluation result can be obtained through weighted averaging, voting, or other combined mechanisms: (twenty one) in, Indicates the first Mapping of evaluation heads, This represents the result fusion function. Because... Given that, therefore each and finally All of these can be further obtained. The purpose of using a multi-evaluation-head approach is to reduce the impact of single-path evaluation bias on subsequent reliability analysis, making the action evaluation results more adaptable to complex visual state changes. Regardless of whether a single evaluation head or a multi-evaluation-head approach is used, the technical essence is the same: to generate action evaluation results that can be compared with the original action labels based on discrete latent action representations.

[0041] In motion assessment results and predict action labels After obtaining the results, step S120 further compares the action evaluation results with the original action labels in the offline sample set. The relationship between these factors determines the sample reliability and overall contamination levels. The related processing can be further expanded to: determining the sample prediction loss based on the deviation between the predicted action results (which are the outcome of action evaluation) and the original action labels; and determining the sample reliability and overall contamination levels based on the sample prediction loss. Figure 3 In the attached figures, reference numeral 302 indicates the sample deviation generation stage, reference numeral 303 indicates the distribution decomposition stage, reference numeral 304 indicates the sample reliability output stage, and reference numeral 305 indicates the overall contamination level output stage.

[0042] In one embodiment, the sample deviation generation step 302 is based on the action prediction result. With original action tag Calculate the sample prediction loss based on the deviation relationship between them. Their relationship can be expressed as: (twenty two) in, Represents the sample prediction loss function. Indicates sample The corresponding sample prediction loss. Due to the action prediction result... Already represented by discrete latent actions The original action label was derived. The quantity is already known in the offline sample set, therefore It can be obtained from the known quantities. Its function is to quantify the deviation between the action evaluation result recovered from the semantics of state change and the action label of the original record in the sample into a value that can be further statistically modeled.

[0043] In one embodiment, if the original action label Using the one-hot representation, the sample prediction loss function is... The classification cross-entropy form can be used: (twenty three) in, Indicates the original action label In the Unique heat components in each category Indicates the action prediction result In the Predicted components on each category. Due to and It is known, therefore It can be calculated directly. The cross-entropy form can characterize the degree of support the prediction distribution has for the original label: if and Highly consistent, then Smaller; if and If there is a clear inconsistency, then The value is relatively large. Therefore, whether the original action label is compatible with the state change semantics can be determined by... The size is reflected in it.

[0044] In one different implementation, the sample prediction loss function Alternatively, an augmented loss form incorporating class distance constraints can be used. This is possible if a predefined class distance function exists between action classes. Then it can be expressed as: (twenty four) in, Represents the classification cross-entropy term. This represents the weighting coefficient of the category distance item. Indicates the predicted action label With original action tag The category distance between them. , as well as All are known, therefore This process can continue. This method can distinguish between completely irrelevant erroneous actions and the degree of deviation from adjacent semantic actions, making subsequent reliability analysis more detailed. Regardless of whether the base loss or the augmentation loss is used, their common function is to explicitly quantify the deviation relationship between the action evaluation result and the original action label.

[0045] In one embodiment, when the sample prediction loss After calculating the loss for all samples, we can obtain the loss set for the entire sample set: (25) Because of each sample All are known, therefore This also becomes a known quantity. The work to be completed in the second half of step S120 is based on... This involves determining which samples correspond to more reliable original action labels, and assessing the overall extent of action label contamination in the current dataset. At this point, a single... This only reflects the deviation of a single sample. To obtain further results on sample reliability and overall contamination level, it is necessary to analyze the entire dataset at the ensemble level. The statistical distribution of the sample. Therefore, this step does not simply judge the reliability of a sample based on a fixed threshold, but rather conducts further analysis on the sample prediction loss set to form more stable reliability results and overall contamination level results.

[0046] In one embodiment, the sample reliability result and the overall contamination level result can be determined based on the sample prediction loss, that is: the sample prediction loss is distributed and decomposed; the sample cleanliness is determined as the sample reliability result based on the distribution decomposition result; and the overall contamination level result is determined based on the distribution decomposition result. Figure 3 The reference numerals 303, 304, and 305 in the accompanying drawings illustrate this process. Specifically, the distribution decomposition step 303 affects the loss set. Each of the Statistical distribution decomposition is performed to statistically distinguish the low-loss sample group from the high-loss sample group. As a result, samples whose original action labels are more consistent with the semantics of state changes are usually classified as low-loss components, while samples whose original action labels are significantly inconsistent with the semantics of state changes are usually more likely to be classified as high-loss components.

[0047] In one embodiment, the distribution decomposition step 303 can employ a hybrid distribution modeling approach. Let the sample prediction loss be... The overall distribution is from Composed of several statistical components, we have: (26) in, This represents the overall distribution of the sample prediction loss. Indicates the first The weights of each statistical component, Indicates the first Parameters of each statistical component Indicates the first Sample prediction loss under each statistical component The conditional distribution. Due to It is obtained from the aforementioned known quantities, therefore all of them can be further utilized. estimate and This allows these parameters to become new known quantities. In this way, the loss set... It is no longer just an isolated set of numerical values, but is reinterpreted as the result of the superposition of multiple different statistical components, providing a basis for subsequently separating more reliable samples and more suspicious samples from the overall sample set.

[0048] In one embodiment, considering the most common two-component case, the low-loss component can be denoted as... High loss components are denoted as Then we have: (27) in, Indicates the weight of the low-loss component. Indicates the weight of high-loss components. and These represent the parameters of the corresponding components. Since all... These parameters are known, therefore they can be obtained through iterative estimation, maximum likelihood estimation, or other parameter estimation methods. Once the parameters are estimated, the samples can be further calculated. Posterior probability of belonging to low-loss component : (28) in, Indicates sample The corresponding sample cleanliness results. In this embodiment, As a result of sample reliability. Because , , , and All are known, therefore It can be obtained from known quantities. The larger the value, the more likely the sample belongs to the low-loss component, and the higher the consistency between its original action label and state change semantics. The smaller the value, the more likely the sample belongs to the high-loss component, and the more likely its original action label is contaminated.

[0049] In one embodiment, the overall pollution level result This can be obtained directly from the weights of the high-loss components: (29) in, This indicates the overall level of pollution. Because... It has already been estimated by the distributed decomposition step 303, therefore This can be determined directly. At this point, It reflects the proportion or statistical weight of samples that are closer to the high-loss group in the entire sample set. The larger the value, the higher the overall level of contamination of action labels in the current sample set; The smaller the value, the closer the action labels in the current sample set are to a reliable state overall. Thus, step S120 outputs the result at the sample level. Output at the overall level This provides a two-tiered basis for adjusting the original action labels in the next step S130 based on the sample reliability results and the overall contamination level results.

[0050] In one different embodiment, the overall contamination level results It's not always necessary to directly take the weight of the component with the highest loss; you can also consider the sample cleanliness results. Further statistical analysis revealed this. For example, it can be presented in the following form: (30) in, Indicates the number of samples. Indicates sample The corresponding sample reliability results. Because... All are known, therefore It can also be further calculated. This form derives the overall contamination level from the sample-level cleanliness results, and can more directly reflect the inverse vector of the average reliability in the current sample set. Regardless of the definition method used, The purpose of these methods is to characterize the degree of action label contamination in the current sample set at the overall level, so that subsequent action label adjustments no longer rely solely on the local judgment of a single sample, but can simultaneously perceive the global contamination status.

[0051] In one embodiment, step S120 has completed the following derivation: first, based on the discrete latent action representation... Generate motion evaluation results And obtain the predicted action label. ; and then according to With original action tag Calculate the sample prediction loss based on the deviation relationship between them. Subsequently, based on all samples Perform distribution decomposition to obtain sample-level reliability results. And overall pollution level results Through this path, step S120 has transformed the relationship between the action information recovered from the state change semantics and the original action labels recorded in the sample into a statistical basis that can be used for subsequent action label adjustment. Specifically, and It wasn't given out of thin air, but rather along... This chain of reasoning is derived step by step from known quantities. Thus, the action label adjustment performed in the next step S130 is no longer a simple replacement, but a conditional adjustment based on both sample-level and set-level judgments.

[0052] Step S130: Adjust the original action labels in the offline sample set according to the sample reliability results and the overall contamination level results to obtain updated action labels.

[0053] exist Figure 1 In the attached diagram, reference numeral 106 indicates the action label adjustment stage. Figure 3 The parts directly corresponding to step S130 include reference numerals 306 and 307, where reference numeral 306 indicates the action update condition generation stage, and reference numeral 307 indicates the label adjustment execution stage. Step S130 follows the sample reliability result obtained in step S120. Overall pollution level results Motion assessment results and predict action labels .because It characterizes the reliability of a single sample at the sample level, while This characterizes the degree of contamination of the current sample set at the global level; therefore, this step does not affect all original action labels. Instead of applying the same processing to all actions, the original action labels are conditionally adjusted based on the combined effect of sample-level and global-level results, thereby obtaining updated action labels. .

[0054] In one embodiment, step S130, adjusting the original action labels in the offline sample set based on the sample reliability result and the overall contamination level result to obtain updated action labels, can be further expanded as follows: determining action update conditions based on the sample reliability result and the overall contamination level result; when the action update conditions are met, replacing the original action labels with the predicted action labels corresponding to the action prediction results; when the action update conditions are not met, retaining the original action labels; and obtaining updated action labels based on the replaced action labels or the retained original action labels. Thus, the internal logic of step S130 is formed by... and The process begins with an action update condition, then moves to the updated action tag. A complete closed loop.

[0055] In one embodiment, the action update condition generation step 306 can first be based on the overall pollution level result. Generate an updated threshold that is related to the overall contamination level of the current sample set. Their relationship can be expressed as: (31) in, This represents the threshold generation function. This indicates the action update threshold. Because... It has already been derived from known quantities in step S120, therefore The threshold can be further derived from known quantities. Its purpose is to incorporate the overall contamination status of the sample set into the determination process of whether the action label of a single sample needs to be updated. If the overall contamination level... A higher threshold indicates that there may be many erroneous action labels in the current dataset. This can be adjusted accordingly to better trigger updates; if the overall level of contamination... A lower threshold indicates that most action labels in the current dataset are likely still reliable; in this case, the threshold... The update conditions can be adjusted to be more conservative to avoid unnecessary replacement of the originally correct action tags.

[0056] In one embodiment, the threshold generation function A monotonic mapping can be used, for example: (32) in, Indicates the upper bound of the high threshold. This indicates the lower bound of the low threshold, and .because , and Since it is a known quantity, therefore This can be further calculated. This mapping indicates that when the overall pollution level... When the value is smaller, meaning the overall data is more reliable, Closer The conditions for updating action tags are more stringent; when the overall pollution level... When the value is relatively large, meaning the overall data contamination is more pronounced. Closer The conditions for updating action labels are more lenient. In this way, the action update condition is no longer a fixed threshold, but a condition that can adapt to changes in the overall pollution situation.

[0057] In one different embodiment, the action update condition may also not be based on a single threshold. Instead of expressing it, it is achieved through functional decision conditions. For example, it can be based on... , and motion assessment results The confidence level is used to jointly calculate the updated score. : (33) in, This indicates that the function for updating the score generation is being updated. Indicates sample The updated score. Then, according to... The relationship with a reference threshold determines whether to update the original action label. Because... , and All are known, therefore This can be continued. This method is essentially still achieved through... and The joint control over whether action labels are updated is simply expanded in form from a single threshold comparison to a comprehensive score judgment. Therefore, the technical essence of step S130 remains unchanged, namely, the adjustment process of the original action labels is jointly controlled by sample-level and overall-level results.

[0058] In one embodiment, if a threshold is used and sample reliability results In the direct comparison method, the label adjustment execution step 307 can generate updated action labels according to the following rules. : (34) in, Indicates sample The corresponding updated action labels. (Due to the predicted action labels) Original action tags Sample reliability results and threshold All are known, therefore It can be directly obtained from known quantities. Here, we use... Replacement is performed at the time. The rule of retention is applied when sample reliability results are low, indicating poor consistency between the original action label and the semantics of state change. In this case, the predicted action label is preferred. Replace original action tag When the sample reliability result is high, it indicates a strong consistency between the original action label and the semantics of the state change. In this case, it is more likely to retain the original action label. In this way, the original action labels will neither be uniformly replaced on all samples indiscriminately, nor will they remain unchanged when faced with obviously suspicious samples. Instead, they will be updated more reasonably through conditional adjustments.

[0059] In one embodiment, if an updated score is used In this manner, the label adjustment execution step 307 can be further written as: (35) in, This indicates that the score reference threshold has been updated. Because... , , and All are known, therefore It can also be obtained from known quantities. This expression is based on the aforementioned... and The update logic is technically consistent: it constructs a control condition by combining sample-level and global-level information, so that the action label update no longer depends on fixed rules, but changes together with the sample state and the sample set state.

[0060] In a different embodiment, the label adjustment execution step 307 may also employ a soft adjustment method instead of a hard replacement method. Let the original action label be... The corresponding unique heat distribution is denoted as Predicting action results Since it represents a set of action distributions, a fused action distribution can be constructed. : (36) in, Indicates the fusion weight. Indicates sample Corresponding fusion action distribution. Fusion weights. According to and Generate, for example: (37) in, This represents the weight generation function. Because... and It is known, therefore This can be further obtained. Then, based on the distribution of fusion actions... Determine the updated action tags : (38) in, Indicates the distribution of fusion actions in the th order. Components on each action category. Because... It is known, therefore This process can continue. Soft adjustment can create a smoother transition between the original action label and the predicted action result, and it helps to mitigate the abrupt changes caused by hard replacement on some boundary samples. Although the implementations differ, their common function is still to adjust the original action label based on the sample reliability results and the overall contamination level results.

[0061] In one embodiment, the fusion weight It can be in the following form: (39) in, Represents a nonlinear compression function. The weighting coefficients represent the reliability of the sample results. The weighting coefficients representing the overall pollution level. This represents the bias term. Because... , , , and All are known, therefore It can be further derived from the known quantities. When higher and At lower levels, Generally, a larger size indicates a greater tendency to preserve the original action label distribution; when lower and At higher levels, The smaller the value, the more appropriate the motion assessment results should be. The reflected action semantics. In this way, the soft adjustment method has a clear control basis at both the sample level and the global level.

[0062] In one embodiment, step S130 has now completed the process of obtaining the sample reliability results. and overall pollution level results To the updated action tags The derivation is as follows. Reviewing the preceding steps, we can see that step S110 generates a discrete latent action representation based on state transition information. Step S120 according to Generate motion evaluation results and through and The deviation relationship is obtained and Step S130 further... and generate Thus, whether the original action label is replaced, retained, or adjusted in any way is based on a basis derived progressively from the semantics of state changes. Therefore, the updated action label... No longer simply using the original action tags The label is not an arbitrary new label given without considering changes in the sample state, but rather is determined by evidence of state changes, action evaluation results, and the overall statistical results of the sample set. This result will be used to correct the offline sample set in the next step, S140.

[0063] In one embodiment, the action update condition generation step 306 can further refine the update process by incorporating the representation stability of samples in the discrete latent space. For example, the discrete latent action representation is output in the discretization encoding step 206. At the same time, it can also output the prototype attribution confidence score. Used to reflect continuous latent motion representation Representation of Discrete Latent Actions The degree of fit between them. Their relationship can be expressed as: (40) in, Indicates sample Discretized confidence values. Due to and The information has already been derived from known quantities in step S110, therefore This can be further calculated. Subsequently, the action update conditions can be derived from... , and Jointly decided. For example, a comprehensive adjustment index can be constructed. : (41) in, , and These represent the weighting coefficients for the sample reliability result, the discretized confidence level, and the overall contamination level result, respectively. Because , , And all weighting coefficients are known, therefore It can be obtained by continuing to calculate from the known quantities. If If the value is below the set threshold, it indicates that the reliability of the original action label for the sample is low, the discrete latent semantic support is weak, or the overall pollution level is high. In this case, it is more suitable to perform action label updates; conversely, it is more suitable to retain the original action label. This method shows that the action update conditions on which step S130 depends can be further expanded around multiple dimensions such as sample level, set level, and latent semantic stability, but its fundamental logic is still to adjust the original action label based on the sample reliability results and the overall pollution level results.

[0064] In one embodiment, if based on a comprehensive adjustment index To perform a tag update, it can be written as: (42) in, This indicates the threshold for comprehensive adjustment of indicators. Because... , , and All are known, therefore This can be further calculated. This method further implements the action update condition into a comparison rule of a comprehensive index. Therefore, step S130 has a certain degree of flexibility in its specific implementation and can be extended according to the characteristics of the visual task, the characteristics of the sample distribution, and the differences in the action space structure.

[0065] Step S140: Correct the offline sample set according to the updated action labels.

[0066] exist Figure 1 In the attached diagram, reference numeral 107 indicates the sample set correction step. Step S140 follows the updated action labels output in step S130. and the original offline sample set received and retained in step S110 Since step S130 only completed the adjustment at the single-sample action label level and has not yet rewritten these updated action labels back into the sample set, it is necessary to utilize [the updated action labels] in step S140. The original offline sample set is then corrected. In other words, while the previous steps identified and generated more reliable action labels, this step uses these more reliable action labels to correct the structure of the sample set itself.

[0067] In one embodiment, correcting the offline sample set based on the updated action labels may include: replacing the original action labels in the offline sample set with the updated action labels, while retaining the observation data, reward information, and next-time observations in the offline sample set, to generate a corrected offline dataset. Let the corrected sample set be denoted as... Then we have: (43) in, This represents the corrected set of offline samples. Indicates sample The corresponding updated action tags. Because... , and It is known when receiving the offline sample set in step S110 that... In step S130, the result has been derived from known quantities, therefore These known quantities can be directly reconstructed. Thus, the action label field in the original offline sample set has been replaced by the updated action label, while the observation data, reward information, and next-time observation retain the content of the original sample structure, thereby forming a corrected sample set suitable for subsequent training.

[0068] In one embodiment, step S140 serves not only to obtain a new data set notation, but also to re-establish the combined relationship of observation data—action label—reward information—next moment observation in the offline sample structure based on a more consistent action label. In other words, if only step S130 yields... If the incorrect action labels are not rewritten into the sample set, the subsequent training phase will still need to deal with the erroneous action labels in the original sample set, and the previous action evaluation and label adjustment will not be able to truly benefit the training input level; while when Once written back to the sample set, the sample structure read for subsequent training will no longer be the same. , but Therefore, this step constitutes a key connection between the preceding action recovery chain and the subsequent training chain.

[0069] In one embodiment, the corrected offline sample set obtained in step S140 It can be directly used as input for subsequent offline reinforcement learning training. For example, it can be used... Based on this, robust training is continued to obtain the target strategy. Figure 4 An additional processing chain is illustrated, where reference numeral 401 indicates a sample correction input stage, reference numeral 402 indicates a value learning stage, reference numeral 403 indicates a constraint enhancement stage, and reference numeral 404 indicates a policy extraction stage. Because... The action labels in the dataset have been adjusted based on state change semantics, sample reliability results, and overall contamination levels. Therefore, the state-action samples received in subsequent training phases have higher consistency with the original samples, thus providing a more reliable data foundation for subsequent value estimation and policy learning. It should be noted that... Figure 4 The processing link is built on the basis that the offline sample set correction has been completed in step S140, without changing the main processing logic composed of steps S110 to S140.

[0070] In one embodiment, the embodiment consisting of steps S110 to S140 can achieve the following technical results: Step S110 generates discrete latent action representations based on state transition information in the offline sample set, so that action semantic recovery is based on the state change relationship between the observed data and the next time-instance observation, rather than directly on the original action label itself; Step S120 generates action evaluation results based on the discrete latent action representations, and determines the sample reliability results and overall contamination results based on the relationship between the action evaluation results and the original action labels, so that the consistency between the original action labels and the state change semantics can be quantified and transformed into sample-level and set-level analysis results; Step S130 adjusts the original action labels based on the sample reliability results and the overall contamination results to obtain updated action labels, so that the update of action labels is no longer an indiscriminate replacement, but is based on the combined effect of sample local reliability and overall contamination status; Step S140 corrects the offline sample set based on the updated action labels, so that the correspondence between the observed data, action labels, reward information and the next time-instance observation in the sample set is more consistent than that of the original samples. Through the above continuous derivation, the sample semantic mismatch problem caused by discrete action label contamination is mitigated, which is conducive to improving the stability and effectiveness of subsequent policy learning.

[0071] Based on steps S110 to S140, further processing can be added to achieve more refined technical effects. In one embodiment, step S140 yields the corrected offline sample set. Subsequently, robust training can be performed based on the corrected offline sample set to obtain the target policy. The relevant processing can be expanded as follows: replace the original action labels in the offline sample set with updated action labels, while retaining the observation data, reward information, and next-time observation from the offline sample set to generate a corrected offline dataset; perform robust training based on the corrected offline dataset to obtain the target policy. Since this additional processing is built upon the correction results formed in steps S110 to S140, its input samples already have a more consistent action label foundation, and the starting conditions for subsequent value estimation and policy learning are correspondingly improved.

[0072] In one embodiment, Figure 4 The calibration sample input stage 401 receives the calibration offline sample set output in step S140. For ease of subsequent description, it can be... Represented as: (44) in, Indicates sample The corresponding updated action tags. Because... The result is already obtained from step S140 by combining known quantities, therefore exist Figure 4 The inputs shown in the link can be directly considered as known inputs for subsequent training. (Compared to the original sample set) compared to, The action labels in the data are no longer the original record labels. Instead, it is the label derived, analyzed, and adjusted through steps S110 to S130. .thus, Figure 4 The subsequent training phase is not based on the original contaminated labels, but on the corrected action labels.

[0073] In one embodiment, Figure 4 The value learning step 402 in the middle is based on the corrected offline sample set. Construct the state-action value learning objective required for subsequent training. Assume the observed data... The corresponding state is represented as Observation at the next moment The corresponding successor state is represented as Then it can be expressed as: (45) (46) in, State represents mapping, This indicates the current state. This represents the successor state. Because... and exist As is known in the text, therefore and You can continue to receive rewards. Then, you can refer to the reward information. Discount Factor and successor state representation Construct target value items : (47) in, Represents the state value function. Indicates sample The corresponding target value item. Because... , and Since it is a known quantity, therefore It can also be further derived. Value learning stage 402 then represents the current state. Updated action tags and target value items It learns the value of state and action. Because... The original action tag has been replaced. Therefore, the state-action correspondence used in subsequent training is more consistent with that of the original samples, thereby reducing the value regression bias caused by action label contamination.

[0074] In one embodiment, Figure 4 The constraint enhancement step 403 can further incorporate conservative constraints or outlier suppression during the value learning process to further reduce the impact of residual erroneous samples on the training process. Let the constraint term be denoted as... The overall value learning objective can then be expressed as: (48) in, This represents the value regression loss function. This represents the estimation of the value of a state-action relationship. This represents the coefficient of the constraint term. Because... , and It is known, therefore Further calculations are possible. It should be noted that the constraint enhancement step 403 is not a mandatory component of the main steps of this invention, but rather is added after the corrected sample set has been obtained. Then, additional processing is applied to further robust the subsequent training. This additional processing is sequential with the preceding steps: the preceding steps first improve the semantic consistency of the samples, and the subsequent additional processing applies training constraints on the basis of the more consistent data, thereby further improving the stability of the target policy learning.

[0075] In one embodiment, Figure 4 In the strategy extraction stage 404, the target strategy is output based on the value learning results. If we denote the strategy extraction target as... Then it can be expressed as: (49) in, This represents the sample weight. Based on sample reliability results Determined by value advantage or other training evaluation metrics. Because... and It is known, therefore It can be further constructed. The strategy extraction stage 404 updates the strategy parameters accordingly, ultimately obtaining the target strategy. Therefore, the set of corrected samples obtained in step S140 pass Figure 4 The additional links in the process continue to influence subsequent policy training, so that the results of action label correction not only remain at the data cleaning level, but also further affect the actual policy learning process.

[0076] In one different embodiment, Figure 4 Subsequent training is not limited to server-side devices; it can also be performed by terminal devices, edge computing nodes, or multi-agent collaborative environments. If performed by a terminal device, the corrected offline sample set... It can be used directly for policy training locally; if executed by an edge computing node, the terminal side can be responsible for sample collection and preliminary correction, while the edge side continues to perform value learning and policy extraction; if executed by the server side, steps S110 to S140 and... Figure 4 The training pipeline can be centrally deployed on the server side. Regardless of the deployment method, the common premise remains: steps S110 to S140 have already completed action label adjustment and offline sample set correction, and subsequent training continues based on the correction results. Therefore, changes in deployment do not alter the core processing logic of the present invention.

[0077] In one embodiment, besides Figure 4 In addition to the additional training links shown, several different implementation methods can be further given around the core technical points in steps S110 to S140, so as to more fully reveal the technical essence of the present invention.

[0078] In one embodiment, state transition representation The construction can employ explicit difference enhancement. Besides the aforementioned method of concatenating differences and products, difference features can also be constructed independently. and corresponding features : (50) (51) Then, according to and Co-generate state transition representation : (52) in, This represents the difference-correspondence joint mapping. Because... and It is known, therefore , and All can be obtained. This method places greater emphasis on the coupling relationship between the direction of state change and the corresponding position. For tasks with large visual state changes and complex background changes, it is beneficial to improve the sensitivity of state transition representation to action-related changes.

[0079] In one embodiment, continuous latent action represents The generation can be achieved using a comparison constraint method. Let the sample... and samples The state transition representations are respectively and If their actions have similar semantic meanings, then it is desirable and The distance is closer in the latent space; if the semantic differences in their actions are large, then it is desirable to... and The distance is greater in the latent space. Comparison constraints can be constructed: (53) in, Indicates sample pairs Are they semantically similar tags? This represents the interval parameter. Because... and Depend on and It is derived that, therefore This can be further derived. By introducing this contrast constraint, the continuous latent action representation possesses a clearer semantic aggregation structure before discretization, which is beneficial for subsequent discrete latent action representation. The stable formation of .

[0080] In one embodiment, sample prediction loss Before entering the distribution decomposition stage 303, normalization can be performed to reduce the impact of different batches or sample sizes on the loss scale. Let the minimum and maximum values ​​of the loss among all samples be respectively... and Then the normalized loss can be obtained. : (54) in, This represents a stable term. Because... , , and All are known, therefore This can be continued. Subsequently, the distributed decomposition step 303 can be based on... Instead Perform statistical modeling. The purpose of this is to make it easier to conduct unified analysis of the sample prediction loss distribution under different task scenarios or different training stages, thereby improving the comparability of sample reliability results and overall contamination level results.

[0081] In one embodiment, sample reliability results It is not necessary to obtain the result directly through a single distribution decomposition; a hierarchical estimation method can also be used. For example, it can be obtained first based on the normalized loss. Obtain coarse-grained reliability results Combined with the results of the motion assessment The confidence level yields fine-grained reliability results. Ultimately As a result of sample reliability Their relationship can be expressed as: (55) (56) (57) in, and Represents the reliability estimation mapping at different levels. Indicates the results of motion assessment The confidence level. Due to and It is known, therefore , and All of these can be obtained sequentially. This method illustrates the sample reliability results. Although expressed in a more general form, it can be further refined into a multi-stage reliability analysis process at the specific implementation level, as long as the final result is a sample-level reliability result that can be used for action label adjustment.

[0082] In one embodiment, the overall pollution level result Smooth updates can also be performed on consecutive processing rounds. Let the th round be... The instantaneous pollution level obtained by round estimation is The previous pollution level result was The pollution level result for the current round can be expressed as: (58) in, This represents the smoothing coefficient. Because... , and Since it is a known quantity, therefore The process can continue. When using this smooth update method, the action update conditions in step S130 will not change drastically due to single-round statistical fluctuations, but will gradually adjust as the data distribution changes. Therefore, the action label adjustment process can maintain more stable update behavior in iterative training or batch processing scenarios.

[0083] In one embodiment, the threshold generation process, in addition to outputting an updated threshold, In addition, it can further output the action replacement intensity coefficient. This allows for control over the degree of fusion between the original action label distribution and the action evaluation result distribution in a soft-adjustment approach. For example, it can be represented as: (59) in, Indicates the lower bound of low substitution strength. This indicates the upper bound of high substitution intensity. Because... , and Since it is a known quantity, therefore This can be continued. Then, the weights are merged. It can be further adopted in the following form: (60) because and All are known, therefore This can be further obtained. Therefore, it is evident that the overall pollution level result can influence whether to update the action label not only through threshold methods but also through replacement intensity methods, thereby giving the label adjustment process greater adjustability.

[0084] In one embodiment, Figure 4 In the policy extraction step 404, when constructing the policy training objective, the sample reliability results can be further utilized. and overall pollution level results The samples are weighted so that the statistical results obtained in the preceding action label correction stage can be passed on to subsequent training stages. Let the sample weights be... Adopt the following form: (61) in, Indicates sample Weights in policy training. Because... and It is known, therefore The weights can be further calculated. The larger the value, the more trustworthy the sample is, both at the sample level and the overall level; weight The smaller the value, the more likely the sample is to be affected by residual noise in the current environment. Based on this, the strategy extracts the target. It can be further expressed as: (62) Therefore, the sample reliability results and overall contamination results obtained in the preceding steps not only participate in the action label adjustment process, but can also continue to affect subsequent policy learning, thereby enabling a closer connection between the intermediate results in the entire chain.

[0085] In one embodiment, if subsequent training uses an advantage-weighted approach, the advantage quantity can be further defined. : (63) in, This represents the estimation of the value of a state-action relationship. This represents a state value estimate. Because... and It is known, therefore This can be obtained from the known quantities. Subsequently, a weighted strategy objective can be constructed: (64) in, This indicates the dominant temperature parameter. Because... , , and All are known, therefore It can be further constructed. This method demonstrates that the action labels corrected through steps S110 to S140 not only change the action field in the training samples, but also further affect the estimation of value advantage and the strength of its role in policy training.

[0086] In one embodiment, if subsequent training focuses on suppressing residual outliers, a sample bias quantity can also be constructed based on the value estimation bias. : (65) in, Indicates sample The regression bias in value learning. Due to... and It is known, therefore This can be further calculated. Subsequently, it can be based on... Generate additional suppression weights : (66) in, This represents the deviation suppression coefficient. Because... and It is known, therefore This can be continued. Therefore, the comprehensive weights in policy training can be further written as: (67) In this way, with the front-end action correction already completed, subsequent training can still dynamically reduce the impact of suspicious samples based on residual bias, thereby further improving the stability of the final policy learning.

[0087] In one embodiment, if an internal evaluation is needed to assess the degree of consistency improvement of the sample set before and after action label adjustment, a sample consistency difference measure can also be defined. Let the consistency metric corresponding to the original action label be... The consistency metric corresponding to the updated action label is Then it can be expressed as: (68) in, Indicates sample The amount of consistency improvement before and after the action label update. Due to and All of these can be calculated from the relationship between the action evaluation results and the labels, therefore This can also be further derived. When When the value is greater than zero, it indicates that the consistency between the action label and the semantics of the state change of the current sample has improved after adjustment in step S130; when When the value is close to zero, it indicates that the consistency of the current sample has changed little before and after the update. This internal evaluation metric can be used for adaptive parameter tuning or model selection during training, but its essence still serves the action label correction goal achieved by the preceding main steps.

[0088] In one embodiment, the core processing flow surrounding steps S110 to S140 can be further extracted to extract the crucial action correction link, which can then be deployed as a more focused implementation. For example, the following processing can be performed only on the observation data, next-time observation, and original action labels in the offline sample set: first, a discrete latent action representation is generated based on the observation data and next-time observation; then, an action evaluation result and sample prediction loss are generated based on the discrete latent action representation; subsequently, the sample reliability result and overall contamination level result are obtained based on the sample prediction loss; finally, the original action labels are adjusted based on the sample reliability result and overall contamination level result. This implementation remains logically consistent with the aforementioned steps S110 to S130, but focuses more on action label correction itself at the deployment level. Thus, the core processing flow of this invention can be implemented not only as a complete offline sample correction scheme but also as a more focused action correction sub-flow.

[0089] In one embodiment, the processing chain from steps S110 to S140 can be executed by a single processing entity or by multiple processing entities collaboratively. If executed by a single processing entity, offline sample reception, state transition representation generation, discrete latent action representation generation, action evaluation, sample reliability analysis, action label adjustment, and sample set correction are all completed within the same device. If executed collaboratively by multiple processing entities, the first processing entity completes observation encoding and state transition representation generation, the second processing entity completes action evaluation, sample reliability analysis, and action label adjustment, and the third processing entity completes sample set correction and subsequent training. Regardless of the deployment method, the result generated in the previous step is still used as the input for the next step, thus the logical structure of the entire processing chain remains unchanged. For the timing relationship of multi-processor collaborative execution, please refer to [reference needed]. Figure 6 .

[0090] In one embodiment, if the terminal device performs part of the preprocessing, then the terminal device can collect observation data. and the next moment observation and perform observation coding mapping. Thus, the observed features are obtained. and subsequent observation features Subsequently, these intermediate features can be sent to edge nodes or server devices, which will then continue to perform state transition representation generation, continuous latent action mapping, discretization encoding, action evaluation, sample reliability analysis, and action label adjustment processes. If this deployment method is adopted, the intermediate quantities that need to be passed between processing entities will include at least... , , , , , , and Since these intermediate quantities can all be obtained from the preceding steps, collaborative deployment only changes the computation location and does not change the technical logic of this invention.

[0091] In one embodiment, if partial action correction is performed by an edge node, the edge node can receive intermediate features or original samples uploaded by the terminal device, complete the main derivation process in steps S110 to S130, and output the updated action label. The server equipment then receives further information. A corrected offline sample set is constructed by combining the retained observation data, reward information, and the next time-step observation. Then, subsequent training is performed. This allows the sample semantic correction stage to be deployed closer to the data source, while the training stage can be deployed in a location with more abundant computing resources. This approach helps balance real-time performance and training complexity in resource-constrained environments.

[0092] In one embodiment, if all processing is centrally performed by the server, then the offline sample set... It can be directly uploaded to the server device, and the server device completes steps S110 to S140 in a unified environment. Figure 4 The attached training link is shown. This method is the most direct in terms of processing links, and intermediate quantities do not need to be split and transmitted across devices, making it suitable for centralized analysis and training of large-scale offline sample sets. Since all processing is completed on the same device or the same group of devices, parameter sharing, batch processing, and multi-round iterative updates are also easier to achieve. Even so, the essence of this invention still lies in the logic of recovering action semantics based on state transition information and adjusting action labels in combination with sample reliability results and overall contamination results, without relying on a specific deployment method.

[0093] In one embodiment, if the method flow consisting of steps S110 to S140 is presented graphically, then Figure 1 A main flowchart can be used, with action steps as the main thread; Figure 2 A flowchart for generating discrete latent action representations can be generated using state transition information. Figure 3 A flowchart for motion assessment, distribution decomposition, and label adjustment can be used; Figure 4 A flowchart of subsequent training driven by calibration samples can be used. These accompanying figures, along with the text in this embodiment, provide a clearer view of the data connections between each step. Specifically, Figure 1 101 to 107 in the table can correspond to offline sample reception, state transition related information extraction, discrete latent action representation generation, action evaluation, reliability analysis, label adjustment, and sample set correction, respectively. Figure 2201 to 206 in the text correspond to the internal links that generate discrete latent action representations based on the observed data and the observation at the next time step; Figure 3 301 to 307 in the text correspond to the internal link that generates motion evaluation results from discrete latent motion representations and further generates updated motion labels; Figure 4 In the diagram, 401 to 404 correspond to additional processing links that execute subsequent training starting from the corrected offline sample set.

[0094] In one embodiment, if the aforementioned processing procedure is further summarized from a data flow perspective, its logic can be described as: offline sample set First, provide observational data. Original action tags Reward Information And the next moment of observation Subsequently, observation data Observation at the next moment The state transition representation is generated by transforming the observation-encoding mapping and state transition representation into a state transition representation. Then, through latent action mapping and discretization mapping, it is transformed into a discrete latent action representation. Discrete latent action representation The motion assessment results are further obtained through motion assessment mapping. and the original action tag Co-generated sample prediction loss The set of predicted losses for all samples The sample reliability results are then obtained through distribution decomposition. and overall pollution level results ;based on and Combined with predicted action labels Generate updated action tags Finally, the updated action tags With retained observation data Reward Information and the next moment observation Jointly construct the corrected offline sample set Therefore, the unknowns obtained in the previous step are continuously transformed into knowns in the next step, resulting in a continuous derivation rather than discrete piecing together of the entire technical solution. For key data flow relationships, please refer to... Figure 5 .

[0095] In one embodiment, the method flow consisting of steps S110 to S140 does not exclude other conventional processing steps. For example, in receiving an offline sample set... Next, the samples can be deduplicated, shuffled, loaded in batches, or indexed; then, observation features can be generated. and subsequent observation features Afterwards, regular normalization, dimension alignment, or batch normalization can be added; this is then used to construct the corrected offline sample set. Subsequently, additional index reconstruction, sample caching, or partitioned storage operations can be performed. These processes are conventional implementation methods that complement the main inventive point of this invention, and their existence does not change the core logic of this invention, which revolves around restoring action tags based on state transition semantics and correcting the offline sample set accordingly.

[0096] In one embodiment, to understand the technical effects of the present invention from the perspective of feasibility, the derivation can be made along the causal chain formed by steps S110 to S140: Step S110 generates discrete latent action representations through state transition related information, enabling action semantic recovery to be based on evidence of state changes; Step S120 further transforms the discrete latent action representations into action evaluation results, and obtains sample reliability results and overall contamination results through the relationship between the original action labels and the action evaluation results, providing a quantitative basis for whether the original action labels are credible; Step S130 then generates updated action labels based on the sample reliability results and overall contamination results, changing the action label update process from fixed replacement to conditional adjustment; Step S145 finally rewrites the updated action labels back into the offline sample set, making the state-action correspondence used in subsequent training more consistent with the original samples. Thus, the interference of discrete action label contamination on subsequent policy learning is reduced, and the sample semantic mismatch problem is alleviated, which is consistent with the technical effects described in the invention.

[0097] In one embodiment, the aforementioned method flow can be further developed into an implementation process more focused on the data processing entity. For example, offline sample reception, state transition information extraction, action evaluation, and tag adjustment can be considered as different execution stages of the same data processing entity. This data processing entity can then complete steps S110 to S140 by sequentially executing sample reception instructions, state change modeling instructions, action evaluation instructions, reliability analysis instructions, and tag update instructions. This approach is closer to a software program or service flow at the implementation level, but its processing logic remains consistent with the aforementioned method embodiment. Subsequent system embodiments and electronic device embodiments can be further developed based on this.

[0098] In one embodiment, the main processing components that play a key role in the method flow comprised of steps S110 to S140 can be further extracted to form an implementation method that focuses more on the action correction process. This implementation method is no longer described from the perspective of the complete offline sample correction-training chain, but rather focuses more on the discrete action label correction itself. Its processing is still based on the logical foundation of the aforementioned steps S110 to S140, but at the narrative level, the emphasis is placed on the main line of action correction: recovering action semantics from state transition information, judging sample reliability from action semantics, and then controlling label updates by sample reliability and overall contamination level.

[0099] In one embodiment, the processing procedure may include the following steps: first, receiving observation data, original action labels, and next-time observations from an offline sample set; then, generating a state transition representation based on the observation data and next-time observations, and further obtaining a discrete latent action representation; subsequently, generating an action evaluation result based on the discrete latent action representation, and obtaining a sample prediction loss based on the relationship between the action evaluation result and the original action label; then, obtaining a sample reliability result and an overall contamination level result based on the sample prediction loss; finally, obtaining an updated action label based on the sample reliability result and the overall contamination level result. Since each step in the above processing flow has been fully elaborated in the preceding embodiments, this implementation can more compactly embody the core action label adjustment logic of the present invention.

[0100] In one embodiment, let the subset of samples received by the more focused processing entity be denoted as: (69) in, This represents the subset of samples used for motion correction. This indicates the number of samples in the subset. Compared to the complete offline sample set, this subset may not explicitly contain reward information. This is because the local processing of the action label correction subject mainly relies on observation data. Original action tags And the next moment of observation The relationship between them. Due to , and Since all of these are known quantities, the local motion correction subject can directly revolve around these three types of information.

[0101] In one embodiment, the processing entity can first obtain the state transition representation through the observation encoding and state transition representation generation process. , and Then, discrete latent action representations are obtained through latent action mapping and discretization mapping. These intermediate processing quantities are consistent with the definitions in the aforementioned embodiments, and therefore will not be repeated individually. Subsequently, the processing entity... Generate motion evaluation results and predict action labels and based on and Calculate the sample prediction loss based on the deviation relationship between them. If the basic cross-entropy form is used, the aforementioned formula (23) still applies; if the category distance enhancement form is used, the aforementioned formula (24) still applies. Therefore, the action correction subject can perform actions without directly relying on reward information. In this case, first complete the local judgment on the credibility of the original action label.

[0102] In one embodiment, the action correction agent further constructs a local loss set based on the sample prediction loss of all samples: (70) Due to each All are known, therefore This becomes the known set in the local motion correction process. Subsequently, it is possible to... Perform local distribution decomposition to obtain local sample reliability results. Results of local and overall pollution levels For example, the same mixed distribution modeling approach as described above can be used: (71) in, This indicates the number of statistical components in the local distribution decomposition. Indicates the first The weights of each statistical component, Indicates the first The parameters of each statistical component. Because... Each of the Given this, further estimation is possible. and This makes these parameters known quantities for the local motion correction subject.

[0103] In one embodiment, if a two-component model with low-loss and high-loss components is still used, the local sample reliability result... It can be represented as: (72) Meanwhile, the results of the overall local pollution level It can be represented as: (73) because , , , and All are known, therefore and It can be obtained from known quantities. Subsequently, the motion correction subject can be based on... , and predict action labels Generate action tags after partial update For example, it can be represented as: (74) in, Indicates according to The generated local update threshold. Because... , , and All are known, therefore It can be obtained directly. Therefore, this more focused processing entity can independently complete action label correction without relying on subsequent value learning or policy learning processes to obtain more reliable action labels first.

[0104] In one embodiment, this more focused processing entity can be deployed as an offline sample preprocessing service. Its input can be observation data from the original offline sample set, original action labels, and the next time-step observation; its output can be updated action labels, or further, a corrected subset of samples with updated action labels. If the updated action labels are used as the output, it can be expressed as: (75) in, This represents the updated set of action labels output by the local action correction entity. Due to the various... All are known, therefore This becomes a set of results that can be directly used by other processing entities. If a subset of samples with correction labels is further output, it can be represented as: (76) In this way, the results completed by the action correction entity can not only serve as intermediate results in the complete offline sample set correction process, but also be deployed directly in the offline sample cleaning process as a separate data preprocessing service.

[0105] In one embodiment, this more focused motion correction subject can also be combined with the complete training chain. That is, the local motion correction subject outputs first. or Then, another processing entity combines the reward information. Complete the sample structure to form a complete set of calibration samples. This information is then used to perform subsequent training. In this way, action label correction and policy learning can be performed centrally by the same processing entity, or collaboratively in stages by different processing entities. The former facilitates unified management of intermediate data and sharing of model parameters, while the latter allows for more flexible splitting of data cleaning and training tasks in engineering deployments.

[0106] In one embodiment, to enhance the adaptability of the motion correction subject to different motion spatial structures, it is also possible to predict motion labels. Action neighborhood constraints are introduced during the generation process. Let the original action label be... The corresponding set of neighborhood actions is Then, a correction label can be selected from the neighborhood action set: (77) in, This represents the predicted action label selected from the neighborhood of the original action label. Because... and It is known, therefore and This can be continued. If this method is adopted, the label update rule in step S130 can be further written as: (78) In this way, the action label update process is controlled not only by the sample reliability and overall contamination levels, but also by the structure of the action space. For discrete action spaces with obvious adjacency or hierarchical relationships, this approach can reduce the probability of large-scale substitutions across semantic boundaries.

[0107] In one embodiment, if the action space does not have an explicit neighborhood structure, then It can degenerate into the entire set of actions, at which point formulas (77) and (78) logically revert to the aforementioned update form without neighborhood constraints. Therefore, the action neighborhood constraint can be regarded as an adaptive extension of the action label update process, rather than a change to the core technical logic of this invention.

[0108] In one embodiment, the sample reliability results can also be considered. Results of overall pollution levels The relationship between the dynamic update strength index This describes the degree of update that the current sample should receive relative to the overall environment. Let... It can be represented as: (79) in, Indicates sample The intensity index is dynamically updated. Due to... and It is known, therefore It can be further determined. The larger the value, the more unreliable the current sample is at the local level and the higher the overall environmental pollution level. In this case, it is more necessary to update the action label. The smaller the value, the more reliable the current sample is locally or the lower the overall environmental pollution level, making it more suitable to retain the original action labels. Using this index, the fusion weights in the soft-adjustment method can be further expressed as: (80) In this way, the adjustment range of the action label directly corresponds to the multiplicative relationship between the sample reliability result and the overall contamination degree result.

[0109] In one embodiment, the output generated by the aforementioned action correction subject can not only be a simple updated action label, but also include auxiliary information such as output sample reliability results, overall contamination level results, and action evaluation results, so that subsequent processing subjects can reuse it. Let the extended output result set be denoted as... Then it can be expressed as: (81) in, This represents the extended output of the motion correction subject. Because... , , and All are known, therefore It can be directly constructed. This method allows the subsequent training subjects to not only receive the updated action labels when receiving the correction results, but also to simultaneously obtain the statistical basis corresponding to the label adjustment process, thereby further realizing the weighting, filtering or constraint of the training process.

[0110] In one embodiment, if the subsequent training subject directly receives the extended output Then train weights It can be constructed directly from the output of the motion correction subject, instead of being obtained through separate calculations. For example, it can be represented as: (82) in, This represents the training weight generation function. Because... and It is known, therefore This process can be continued. In this way, the coupling between the preceding action correction subject and the subsequent training subject is further strengthened, indicating that the action label correction in this invention is not only an independent data preprocessing process, but can also serve as an important source of information for subsequent robust training.

[0111] In one embodiment, to further demonstrate the system-level implementation of the method of the present invention, Figure 7 A discrete action correction system for visual offline reinforcement learning can be illustrated. The system includes a latent action generation module 701, an evaluation module 702, a label adjustment module 703, and a sample correction module 704.

[0112] The latent action generation module 701 is used to acquire an offline sample set and generate a discrete latent action representation based on the state transition information in the offline sample set. In one embodiment, the latent action generation module 701 may further include an observation receiving unit, a feature encoding unit, a state transition representation unit, a continuous latent action mapping unit, and a discretization encoding unit. The observation receiving unit is used to receive observation data and the next time-instance observation; the feature encoding unit is used to generate observation features and subsequent observation features respectively; the state transition representation unit is used to construct a state transition representation based on the observation features and subsequent observation features; the continuous latent action mapping unit is used to generate a continuous latent action representation based on the state transition representation; and the discretization encoding unit is used to discretize the continuous latent action representation to obtain a discrete latent action representation. The data transmission relationship between the units is consistent with the textual derivation in step S110, therefore, the latent action generation module 701 can be regarded as the functional counterpart of step S110 at the system level.

[0113] The evaluation module 702 is used to generate action evaluation results based on discrete latent action representations, and to determine sample reliability results and overall contamination level results based on the relationship between the action evaluation results and the original action labels in the offline sample set. In one embodiment, the evaluation module 702 may further include an action evaluation generation unit, a deviation generation unit, a distribution decomposition unit, a sample reliability output unit, and an overall contamination level output unit. The action evaluation generation unit corresponds to... Figure 3 301 in the code is used to generate motion evaluation results based on discrete latent motion representations; the deviation generation unit corresponds to... Figure 3 302 in the diagram is used to obtain the sample prediction loss based on the deviation between the action evaluation result and the original action label; the distribution decomposition unit corresponds to... Figure 3 303 in the formula is used to perform statistical decomposition on the sample prediction loss set; the sample reliability output unit corresponds to... Figure 3 304 in the output is used to output sample reliability results; the overall contamination level output unit corresponds to... Figure 3305 in the equation is used to output the overall pollution level result. Therefore, the evaluation module 702 undertakes all the main processing logic of step S120 at the system level.

[0114] The label adjustment module 703 is used to adjust the original action labels in the offline sample set according to the sample reliability results and the overall contamination level results, to obtain updated action labels. In one embodiment, the label adjustment module 703 may include an action update condition generation unit and a label adjustment execution unit. The action update condition generation unit is used to generate update thresholds, update scores, fusion weights, or other forms of action update conditions according to the overall contamination level results; the label adjustment execution unit is used to generate updated action labels according to the sample reliability results, action update conditions, action evaluation results, and original action labels. This module corresponds to the aforementioned steps S130 and... Figure 3 The implementation process of 306 and 307 can be any one of the forms or combinations shown in formulas (31) to (42).

[0115] The sample correction module 704 is used to correct the offline sample set according to the updated action labels. In one embodiment, the sample correction module 704 may include a label replacement unit and a sample recombination unit. The label replacement unit is used to replace the original action labels in the offline sample set with the updated action labels, and the sample recombination unit is used to retain the observation data, reward information, and next-time observation in the original offline sample set, and combine them with the updated action labels to generate a corrected offline sample set. This module corresponds to step S140 and the processing procedure expressed by formula (43). Thus, Figure 7 The system architecture described above can fully cover the main processing flow in the aforementioned method embodiments.

[0116] In one embodiment, the system implementation does not require that each module exist as a physically independent module. The latent action generation module 701, evaluation module 702, label adjustment module 703, and sample correction module 704 can be deployed on the same processor as software functional modules, or they can be distributed across multiple processing nodes as service-oriented components. As long as the data flow relationship between these modules remains consistent with the aforementioned method embodiment, that is, first generating discrete latent action representations, then generating action evaluation results and determining sample reliability results and overall contamination level results, then generating updated action labels, and finally correcting the offline sample set, all of these fall under the implementation of this invention.

[0117] In one embodiment, Figure 8An electronic device 801 for implementing the aforementioned discrete motion correction method may be shown. The electronic device 801 may include a processor 802, a memory 803, a communication interface 804, and a bus 805. The processor 802, memory 803, and communication interface 804 are respectively connected to the bus 805 to achieve data interaction via the bus 805. The memory 803 is used to store program instructions and related data; the processor 802 is used to execute the program instructions stored in the memory 803 to implement all or part of the steps in the aforementioned method embodiments; and the communication interface 804 is used to receive an offline sample set or output a corrected offline sample set and related intermediate results to an external device.

[0118] In one embodiment, when the processor 802 executes the program instructions, it acquires an offline sample set and generates a discrete latent action representation based on state transition information in the offline sample set; it generates an action evaluation result based on the discrete latent action representation, and determines a sample reliability result and an overall contamination level result based on the relationship between the action evaluation result and the original action labels in the offline sample set; it adjusts the original action labels in the offline sample set based on the sample reliability result and the overall contamination level result to obtain updated action labels; and it corrects the offline sample set based on the updated action labels. Thus, the program logic executed by the electronic device 801 is consistent with the main steps in the aforementioned method embodiment.

[0119] In one embodiment, the electronic device 801 can be deployed as a terminal device, an edge computing node, a server device, or any computing node in a cloud-edge collaborative system. If deployed as a terminal device, the device can locally complete offline sample reception, state transition information processing, and action label correction. If deployed as an edge node, the device can receive samples or intermediate data uploaded by the terminal device and complete the main processing steps of action label correction. If deployed as a server device, all processing from offline sample correction to subsequent training can be centrally completed. Since these different deployment methods do not change the main steps of program execution, they do not change the core technical content of this invention.

[0120] In one embodiment, the present invention can also be implemented as a computer-readable storage medium. This computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform all or part of the steps in the aforementioned method embodiments. The computer-readable storage medium can be a read-only memory, random access memory, flash memory, hard disk, solid-state drive, optical disk, or other non-transitory storage medium. Since the intermediate quantity derivation relationships in the method embodiments have been gradually given in the specification, the storage medium implementation only needs to carry the program instructions corresponding to these processing procedures to enable the processor to complete the corresponding calculations.

[0121] In one embodiment, to further summarize the technical solution of the present invention from the perspective of the overall chain, the solution can be understood as a continuous data derivation chain from state change semantic recovery to action label update to sample structure reconstruction. Specifically, the observation data in the offline sample set and the next-time observation are first used to recover the state change semantics. The recovered state change semantics are then transformed into discrete latent action representations, which are further transformed into action evaluation results. The relationship between the action evaluation results and the original action labels is then quantified as sample prediction loss. The sample prediction loss is further transformed into sample reliability results and overall contamination results. Finally, the sample reliability results and overall contamination results control the adjustment of the original action labels, and the adjusted results are written back to the offline sample set. Thus, the sample structure that originally contained semantic mismatch problems is reconstructed into a more consistent sample structure, thereby providing a more reliable data foundation for subsequent training.

[0122] In one embodiment, from the perspective of the derivation process of the technical effect, the above-mentioned continuous data derivation chain can achieve the following effects: Since action semantic recovery is based on the observed data and the state change information reflected by the next moment's observation, the discrete category-level errors in the original action labels will not be directly inherited in the action semantic recovery stage; Since the deviation relationship between the action evaluation result and the original action label is further transformed into the sample reliability result and the overall contamination degree result, the action label update process no longer relies on a single prediction of a single sample, but simultaneously refers to the statistical basis at the sample level and the set level; Since the updated action label is rewritten back to the offline sample set, the state-action correspondence used in subsequent training is more consistent than that of the original sample. Thus, the sample semantic mismatch problem caused by discrete action label contamination in the visual offline reinforcement learning scenario can be alleviated, and the stability and effectiveness of subsequent policy learning are correspondingly improved.

[0123] In one embodiment, if we further generalize from the correspondence between the accompanying drawings and the text, then... Figure 1 The focus is on the main processing steps. Figure 2 The key feature is the generation process of discrete latent action representations. Figure 3 The key aspects are motion assessment, reliability analysis, and tag update processes. Figure 4 The key feature is the extended application of the corrected sample set in subsequent training. Figure 7 The key feature is the modular correspondence of the methodology and process at the system level. Figure 8 The emphasis is on the hardware implementation of the methodology at the electronic device level. The diagrams are not independent of each other, but rather represent different levels of representation of the same technical solution. Figures 1 to 4 Main expression method logic, Figure 7 Mainly expresses the organization of system functions, Figure 8This primarily describes the form of the equipment carrier. This progressive expression, from method to system to equipment, allows for a more complete support relationship between the abstract and implementation layers of the technical solutions in the specification.

[0124] In one embodiment, if a further understanding of the relationship between the method embodiment, system embodiment, and device embodiment is required, it can be considered that: the method embodiment describes what the technical solution does and why it is processed in this order; the system embodiment describes the organization of these processes at the functional module level; and the device embodiment describes the carrier method of these processes at the hardware and program execution level. Since each module, unit, and execution step in the system embodiment and device embodiment can be directly derived from the aforementioned method embodiment, the system embodiment and device embodiment are technically consistent with the method embodiment, and their differences mainly lie in the perspective of expression rather than the inventive concept itself.

[0125] In one embodiment, from the perspective of parameter settings, the dimension parameters, weight coefficients, upper and lower threshold bounds, smoothing coefficients, temperature parameters, interval parameters, and suppression coefficients appearing in the aforementioned formulas can all be set according to the specific task scenario, sample size, action space size, complexity of state changes, and computational resource conditions. These parameters are not required to be limited to a fixed value, but rather are required to support the aforementioned derivation relationship, make relevant intermediate quantities calculable, and enable the action label adjustment process to be implemented. In this way, the present invention retains sufficient implementation details while avoiding unnecessarily limiting the scope of protection to a specific set of parameter values.

[0126] In one embodiment, from the perspective of flexibility in implementation, the aforementioned observation encoding mapping, state transition representation generation mapping, latent action mapping, discretization mapping, action evaluation mapping, loss function, distribution decomposition method, action update condition generation method, and subsequent training method can all be implemented using different structures, different algorithms, or different deployment locations. As long as these different implementation methods still revolve around the same core logic, namely, generating discrete latent action representations based on state transition-related information in the offline sample set, generating action evaluation results based on discrete latent action representations, determining sample reliability results and overall contamination level results based on the relationship between action evaluation results and original action labels, adjusting the original action labels based on sample reliability results and overall contamination level results, and correcting the offline sample set based on the updated action labels, this can be considered an implementation of the present invention.

[0127] In one embodiment, the relationships between the variables in the foregoing embodiments can be further summarized. Observation data and the next moment observation It is the initial known input; observed features Subsequent observation characteristics State transition representation Continuous latent motion representation and discrete latent action representation It is an intermediate quantity derived along the semantic recovery link of state changes; action evaluation result Predicting action tags Sample prediction loss Sample reliability results and overall pollution level results It is an intermediate quantity derived along the action evaluation and reliability analysis chain; the updated action label It is the result obtained by adjusting the link along the label; the corrected offline sample set. It is the structural result obtained along the sample reconstruction link; target value item Training objectives Strategic Objectives Other additional training amounts are based on This is the basis for subsequent training. Therefore, the variables in the specification are not defined in isolation, but rather are the result of a step-by-step unfolding strictly along the data flow.

[0128] In one embodiment, from the perspective of the closed-loop nature of the overall solution, this invention does not merely perform a static replacement on offline samples before they enter training, but rather forms a complete chain encompassing action semantic recovery, sample reliability analysis, action label adjustment, sample set correction, and subsequent training applications. In this chain, the action label contamination problem is first explicitly exposed through state change semantic recovery, then explicitly quantified through sample reliability results and overall contamination levels, followed by the updated action labels being written back to the offline sample set, and finally further applied to subsequent training through the corrected offline sample set. Thus, a continuous relationship is formed between front-end data correction and subsequent training processes, enabling this invention to form a relatively complete technical solution path for the discrete action label contamination problem in visual offline reinforcement learning scenarios.

[0129] In one embodiment, if the invention is understood from a more general application perspective, the solution is not limited to a specific task. For robot navigation scenarios, the observation data and the next-moment observation can come from visual navigation camera sequences, and the original action labels can be discrete navigation control commands. For industrial operation scenarios, the observation data and the next-moment observation can come from visual sampling sequences during the operation process, and the original action labels can be discrete operation commands. For video game scenarios, the observation data and the next-moment observation can come from game frames, and the original action labels can be discrete button actions. For other vision-driven offline decision-making tasks, as long as the sample structure can be expressed as the state transition relationship between the observation data, action labels, and the next-moment observation, then the aforementioned steps S110 to S140 and their additional training processing are all valid. Therefore, the invention has good scenario applicability.

[0130] In one embodiment, from an engineering implementation perspective, the offline sample set... Corrected offline sample set and sample reliability results Overall pollution level results Intermediate values ​​can be stored in local cache, edge node database, server memory, or distributed file system, respectively. A one-to-one correspondence can be established between the sample index and observation data, original action labels, updated action labels, reward information, and the next-time observation, thus facilitating sample reorganization in step S140. Sample reliability results. Other intermediate quantities can also be attached as additional fields to the sample index for reuse in subsequent training. Thus, the derivation of intermediate quantities in the aforementioned method embodiments is not only logically sound but also supported in engineering by conventional data structures and caching mechanisms.

[0131] In one embodiment, a device implementation corresponding to the foregoing method embodiment can also be provided. The device may include at least one processor, at least one memory, and a data bus connecting the processor and the memory. The memory stores program instructions, which, when executed by the processor, cause the processor to perform the following sequentially: receiving an offline sample set; generating a discrete latent action representation based on observation data and the next time-instance observation; generating an action evaluation result based on the discrete latent action representation and determining the sample reliability result and the overall contamination level result; generating updated action labels based on the sample reliability result and the overall contamination level result; correcting the offline sample set based on the updated action labels; and continuing subsequent training based on the corrected offline sample set when necessary. Through the above execution sequence, the device implementation can fully implement the aforementioned method flow at the hardware level.

[0132] In one embodiment, all method steps, system modules, and device execution processes of the present invention can be implemented by program instructions controlling related hardware, or by dedicated circuits, programmable logic devices, or a combination thereof. Accordingly, the systems, devices, modules, units, or links in the foregoing embodiments can be implemented in software, hardware, or a combination of both. The division of modules, units, or links is merely a logical division for the convenience of illustrating the technical solution of the present invention; in actual implementation, they can be combined, split, or integrated as needed.

[0133] In one embodiment, the technical solution comprising steps S110 to S140 can be further elaborated from a temporal processing perspective to illustrate how intermediate quantities are progressively determined at different stages. Specifically, in the offline sample set... After being received, the first thing to determine is the observation data. Original action tags Reward Information and the next moment observation Subsequently, in the state change modeling phase, observed features Subsequent observation characteristics State transition representation Continuous latent motion representation and discrete latent action representation These are determined sequentially; then, in the motion evaluation and reliability analysis phase, the motion evaluation results... Predicting action tags Sample prediction loss Sample reliability results and overall pollution level results They are determined sequentially; then, in the label adjustment phase, the updated action labels are... It was determined; finally, in the sample set correction phase, the corrected offline sample set was determined. The process is complete. In this way, every result in the entire process is derived from the result obtained in the previous step. Therefore, this scheme is not a simple superposition of multiple independent steps, but a continuous processing process that derives new known quantities step by step along known quantities.

[0134] In one embodiment, from a batch processing perspective, the processing in steps S110 to S140 can be executed either sequentially for a single sample or in parallel for a batch of samples. Let the set of sample indices for a batch be denoted as... Then the offline sample subset corresponding to this batch can be represented as: (83) in, This represents the set of sample indices in the current batch. Because each sample... , , and All are known, therefore You can directly use the offline sample set The data is extracted from the middle. Subsequently, observation encoding, state transition representation generation, latent action mapping, discretization encoding, action evaluation, sample prediction loss calculation, and label adjustment can all be executed in parallel on this batch. The introduction of batch processing does not change the dependencies between the intermediate quantities, but only improves the implementation efficiency.

[0135] In one embodiment, if a batch processing method is used, the observed features and subsequent observed features can be represented as batch feature matrices, respectively. and : (84) (85) in, Indicates batch The set of observation data in Indicates batch The next time-instance observation set. Because... and All are known, therefore and It can be calculated from known quantities in batches. Furthermore, the batch state transition characterization can be expressed as: (86) in, This represents the state transition representation generation mapping for batch features. Subsequently, the continuous latent action representation and the discrete latent action representation can also be expressed as follows: (87) (88) Therefore, the intermediate quantities related to state transition processing defined for single samples are naturally extended to batch processing scenarios. Since the batch processing method still maintains the main line from state change information to action semantic recovery, it is merely an extension of the implementation form and does not change the technical essence of the present invention.

[0136] In one embodiment, if it is necessary to uniformly calculate the sample prediction loss in a batch processing scenario, a batch loss vector can be defined. : (89) in, This represents the set of predicted losses for the current batch of samples. This is based on the action evaluation results of each sample within the batch. and original action tags All are known, therefore It can be directly calculated from known quantities. Furthermore, if the overall pollution level results are updated using a batch-cumulative update method, then in the [number]th [period]... After batch processing, the estimated value can be used based on the current batch. New overall pollution levels can be obtained from historical estimates: (90) in, This represents the batch smoothing coefficient. Because... , and All are known, therefore This can be further calculated. This method shows that the overall pollution level result can be obtained not only in a single full-volume analysis scenario, but also gradually stabilized in batch processing, streaming processing, or multiple update scenarios.

[0137] In one embodiment, if we further expand on the use of the sample reliability results, then the sample reliability results Besides being used for action label adjustment, it can also be used for sample screening. Let the screening threshold be... Then the highly reliable sample set and low-reliability sample sets They can be represented as follows: (91) (92) because Both the sample quadruple and the sample quadruple are known, therefore and This can be determined from known quantities. In this way, during subsequent training phases, more reliable samples can be prioritized or assigned higher weights, while less reliable samples can be corrected before participating in training, or only participate in training at a lower frequency. This approach further demonstrates that sample reliability results not only serve label updates but can also influence sample organization and training strategies.

[0138] In one embodiment, if for a lower reliable sample set Perform stronger label updates, while applying to higher-reliability sample sets. To perform more conservative tag updates, a two-tier update rule can be defined. For example, for... The samples in the data use a threshold. ,right The samples in the data use a threshold. ,in Accordingly, the action label adjustment rules can be expressed as: (93) because , , , , and All are known, therefore This can be further derived. Through this hierarchical rule, action label adjustment can balance the strong correction requirements for low-reliability samples and the over-correction and suppression requirements for high-reliability samples.

[0139] In one embodiment, if the temporal context of the samples is further considered, the sample reliability result can also participate in the update decision together with the reliability results of samples at adjacent time steps. Let the first... The reliability results of adjacent samples before and after each sample are as follows: and Then a smoothed local reliability result can be constructed. : (94) in, , and The fusion weights represent the reliability results of adjacent samples, and satisfy the following conditions: .because , , And all weighting coefficients are known, therefore This can be further obtained from known quantities. Subsequently, it can be used... Alternative This allows for the adjustment of action labels. In scenarios where there is a clear temporal continuity between sequence samples, the impact of occasional fluctuations in a single sample on label updates can be mitigated.

[0140] In one embodiment, if the next time interval is observed In addition, it can obtain longer time-domain follow-up observation sequences, such as Then, the state transition related information can be extended to a multi-step observation sequence. Let the set of multi-step successor observations be denoted as... Then we have: (95) Subsequently, the multi-step state transition characterization can be expressed as: (96) in, This represents the generation of a multi-step state transition representation map. Because... and It is known, therefore This can be further calculated. Multi-step state transition representation can retain a more complete action influence trajectory in scenarios with strong action aftereffects, thus making the generation of discrete latent action representations more stable. Even so, its technical essence is still an extended implementation of generating discrete latent action representations based on state transition related information.

[0141] In one embodiment, if the observation data itself is already a multi-frame temporal input, the observation coding mapping can first encode a single frame, and then obtain temporal features through temporal aggregation. Let the observation data... Depend on Frame composition, i.e. Then it can be expressed as: (97) (98) in, Represents a single-frame encoding mapping. This represents a temporal aggregation mapping. Since the observations for each frame are known, therefore... and All of these can be further derived. Similarly, the characteristics of subsequent observations can be obtained from the observations at the next time step using the same method. This approach demonstrates that the definition of state transition-related information in step S110 is not limited to a single-frame image, but can be naturally extended to visual temporal input scenarios.

[0142] In one embodiment, if the action evaluation result Besides being used to determine predicted action labels In addition, a measure of the uncertainty of the predicted distribution is needed, which can be used to further define the prediction entropy. : (99) in, Indicates sample The action evaluation prediction entropy. Due to It is known, therefore It can be directly obtained from known quantities. A larger prediction entropy indicates a more even distribution of action evaluation results, and a less certain understanding of the model's judgment on the semantics of the current sample action; a smaller prediction entropy indicates a more concentrated distribution of action evaluation results, and a more definitive understanding of the model's judgment. Based on this, the sample reliability result can also be further corrected by incorporating prediction entropy. For example, an entropy-corrected reliability result can be defined. : (100) in, This represents the entropy suppression coefficient. Because... , and All are known, therefore This can be further calculated. In this way, even if the sample prediction loss is low, it is possible to avoid being overly optimistic and judging the sample as a highly reliable sample when the uncertainty of the action evaluation itself is high.

[0143] In one embodiment, if the reliability result after entropy correction is... For action tag updates, it can be written as: (101) because , , and All are known, therefore This can be further explained. This method further illustrates that the action label update conditions do not have to rely solely on the reliability results of a single sample; uncertainty information related to the action evaluation distribution can also be introduced on top of the sample reliability results.

[0144] In one embodiment, if the distributed decomposition step 303 no longer uses scalar loss... Instead, it is a joint feature vector composed of the sample prediction loss and the prediction entropy. Then it can be expressed as: (102) Subsequently, their joint distribution can be expressed as: (103) in, In the joint distribution, the th The weights of each statistical component, This represents the parameter corresponding to the statistical component. Because... and It is known, therefore It can be further obtained, and thus estimated. and This joint modeling approach can simultaneously utilize the degree of deviation and prediction uncertainty to determine the reliability of samples, and is often more conducive to improving the separation effect of distribution decomposition in complex visual scenes.

[0145] In one embodiment, if the overall pollution level result If local statistics are needed based on time periods or task phases, then the first... Results of local contamination levels for individual sample clusters or time periods : (104) in, Indicates the first The set of sample indices corresponding to each sample cluster This indicates the number of samples in the sample cluster. Because... and each All are known, therefore This can be further calculated. During subsequent label adjustments, the degree of local contamination can be used based on the cluster or stage to which the sample belongs. Alternative global pollution level results This allows the action label updates to better align with the local data distribution.

[0146] In one embodiment, if the result of local contamination level is used Then the threshold is updated locally. It can be represented as: (105) Subsequently, the sample The cluster to which it belongs is When this happens, its update rule can be expressed as: (106) This approach enables action label updates to no longer rely solely on a globally uniform threshold, but to perceive the differences in action label contamination within local sample clusters.

[0147] In one embodiment, if it is desired to use the motion evaluation results With original action tag The deviation between two parts is divided into confidence deviation and category deviation, and the confidence deviation can be defined. and category deviation .For example: (107) in, Indicates the results of motion assessment In the original action tag The components corresponding to the category. Meanwhile, the category deviation can be expressed as: (108) in, This represents the indicator function. Subsequently, combined deviations can be constructed: (109) in, and This represents the combination coefficient. Because... , , and All are known, therefore This can be continued. This method illustrates that the sample prediction loss is not limited to the form of cross-entropy or enhancement loss; it can also be expressed in a more direct form of deviation combination, as long as it can quantify the relationship between the action evaluation result and the original action label.

[0148] In one embodiment, if a sample replay control strategy is adopted in the subsequent training phase, the sample reliability results can be used as a basis for decision-making. Overall pollution level results and value regression bias Construct replay priority : (110) in, Indicates sample Priority in the replay buffer. Because... , , and All are known, therefore This can be further calculated. Therefore, the statistical results obtained in the action label correction stage can be used for sample selection in subsequent training stages, thereby further enhancing the effectiveness of the invention at the training level.

[0149] In one embodiment, if the replay control strategy needs to determine whether a batch of samples should enter the current training round, it can be based on replay priority. With threshold Filter relationships: (111) in, This represents the set of sample indices participating in the training in the current training round. Because... and All are known, therefore This can be further demonstrated. This approach illustrates that a closer technical connection can be formed between front-end action label correction and back-end sample selection.

[0150] In one embodiment, to avoid over-reliance on a single indicator in sample replay, a multi-factor replay score can also be constructed. : (112) in, , and This represents the weight coefficient of each factor. Because... , , And all weight coefficients are known, therefore This can be further calculated. This method integrates sample reliability, overall environment reliability, and regression bias suppression information during the training phase into the sample replay scheduling, thereby further improving the stability of subsequent training.

[0151] In one embodiment, from a lower-level engineering implementation perspective, the intermediate quantities corresponding to the aforementioned formulas can be stored in memory 803 in the form of tensors, vectors, matrices, or scalars, and can be accessed sequentially or in parallel by processor 802. For example, observation datasets... and It can be stored as a batch tensor for observing features. and It can be stored as a feature matrix, representing the state transition. and discrete latent action representation It can be stored as an intermediate representation matrix, the sample reliability result. Overall pollution level results Updated action tags and training weights These can then be stored as additional fields related to the index. Although these storage structures can be implemented in various conventional ways, their data organization directly corresponds to the processing chain in this invention, thus supporting the actual execution of the aforementioned method flow in electronic devices.

[0152] In one embodiment, further described from the perspective of program execution order, the program instructions may include: sample reading instructions, observation encoding instructions, state transition representation generation instructions, latent action mapping instructions, discretization encoding instructions, action evaluation instructions, loss generation instructions, distribution decomposition instructions, update condition generation instructions, label adjustment instructions, sample recombination instructions, and subsequent training instructions when needed. These program instructions are called sequentially or in groups by the processor to complete the calculation process corresponding to the aforementioned formulas. Therefore, this invention is not only feasible at the logical derivation level but also has a clear execution path at the program execution level.

[0153] In one embodiment, if understood from the perspective of avoiding circumvention of the protection scope, then the aforementioned observation coding mapping State transition representation generation mapping Subconscious action mapping Discretization mapping Motion evaluation mapping The distribution decomposition method, label update rules, and subsequent training strategies can all be implemented in different forms, as long as they still satisfy the logical relationship between the aforementioned main steps. That is, recovering action semantics through state transition information, judging sample reliability and overall contamination level through the relationship between action evaluation and original action labels, and then adjusting action labels and correcting the offline sample set accordingly, all fall within the technical concept of this invention. Therefore, the various mappings, constraints, and scoring functions in the foregoing embodiments, while providing a specific implementation method, also reveal multiple alternative and extended implementation directions.

[0154] In one embodiment, as summarized above, this invention forms a complete processing path from data semantic correction to training input reconstruction by introducing a state-change-driven action semantic recovery chain, a reliability analysis chain based on the relationship between action evaluation and original action labels, and an action label adjustment chain jointly controlled by sample-level and global-level results before offline samples enter subsequent training. Thus, the sample semantic mismatch problem caused by discrete action label contamination in visual offline reinforcement learning scenarios can be explicitly identified and processed before training, allowing subsequent training to be built on a more consistent data foundation. This overall technical effect has been elaborated and supported in the aforementioned method embodiments, system embodiments, and device embodiments.

[0155] In one embodiment, the foregoing method embodiments, system embodiments, electronic device embodiments, and computer-readable storage medium embodiments are substantially corresponding to each other. The method embodiment provides a processing flow for discrete action correction around an offline sample set; the system embodiment provides the organization of this processing flow at the functional module level; the electronic device embodiment provides the implementation of this processing flow at the hardware carrier and program execution level; and the computer-readable storage medium embodiment provides the implementation of this processing flow at the program product level. Thus, a multi-layered protection structure from method and system to device and storage medium can be formed around the same inventive concept.

[0156] In one embodiment, if we further narrow down the implementation path of the overall technical solution, it can be considered as follows: First, extract information related to state changes from the offline sample set, and thereby recover discrete action semantics; then, use the relationship between the recovered discrete action semantics and the original action labels to determine the reliability of the samples at the action label level and assess the overall contamination level of the sample set; next, control the adjustment process of the original action labels based on the sample reliability results and the overall contamination level results to obtain updated action labels; finally, use the updated action labels to correct the offline sample set, and further use the corrected sample set for subsequent training when necessary. This implementation path is logically coherent, with clear sequential relationships between intermediate quantities, thus supporting the feasibility of the discrete action correction scheme proposed in this invention.

[0157] In one embodiment, if we further summarize the dependencies between the aforementioned formulas, it can be understood as the following continuous derivation: Formula (1) gives the basic form of the offline sample set; Formulas (2) to (16) give the generation process from the observed data and the next time-time observation to the discrete latent action representation; Formulas (17) to (30) give the generation process from the discrete latent action representation to the action evaluation result, sample prediction loss, sample reliability result, and overall contamination result; Formulas (31) to (42) give the generation process from the sample reliability result and the overall contamination result to the updated action label; Formulas (43) to (68) give the generation process from the updated action label to the corrected offline sample set and the subsequent training related target quantity; Formulas (69) to (112) further give the variant implementation, extended implementation, and engineering implementation methods around the core processing link. Thus, the formulas in this specification are not given in isolation, but rather collectively serve the same continuous technology chain.

[0158] In one embodiment, although the aforementioned various implementation methods differ in formula form, constraint expression, sample organization, and deployment, they all revolve around the same technical essence: action semantics are not directly recovered from the original action labels themselves, but rather an action semantic representation independent of the original action labels is formed using state transition information in the offline sample set; the relationship between this action semantic representation and the original action labels is then used to evaluate sample reliability and overall contamination level; the results at both the sample level and the overall level jointly determine whether and how to update the original action labels; finally, the updated action labels are reorganized through sample set recombination to truly apply to the data structure upon which subsequent training depends. Any implementation method that still revolves around this technical essence can be considered an implementation of the present invention.

[0159] In one embodiment, the aforementioned technical solution can complete the correction of discrete action labels in an offline sample set without introducing online environment interaction. Therefore, it is suitable for application environments where online trial and error costs are high, security risks are significant, or training data mainly comes from existing playback records. Since the action label correction process mainly relies on the relationship between the observation data in the offline samples, the original action labels, and the observation at the next moment, without relying on additional manual relabeling of each sample, it has good practical value in scenarios with a large number of offline samples. Furthermore, because this solution combines sample-level reliability analysis with overall contamination level assessment, it is more conducive to maintaining stable action label adjustment behavior in data environments with different contamination levels compared to methods that rely solely on a single sample to determine whether to update the action label.

[0160] In one embodiment, the various embodiments described in the specification are only used to explain the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Equivalent substitutions, directly conceived transformations, or combinations of conventional technical means made by those skilled in the art based on the disclosure of the present invention regarding the aforementioned mapping functions, constraint functions, threshold generation methods, sample grouping methods, module deployment methods, hardware structures, or program organization methods should all fall within the scope of protection of the present invention.

[0161] It should be understood that the specific embodiments described in the specification are merely illustrative examples, and those skilled in the art can adjust the order of steps, module division, formula form, parameter settings, deployment location, or implementation structure without departing from the technical concept of the present invention. Regarding the order of steps, some steps can be executed in parallel, alternately, or in combination, provided that data dependencies are satisfied. Regarding module division, some modules, units, or links can be combined, split, or integrated. Regarding parameter settings, each parameter can be adjusted according to the specific task environment, sample size, action space size, and state change complexity without affecting the core technical content of the present invention.

[0162] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, or combinations made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0163] It should be understood that the method steps in the foregoing embodiments can be implemented by program instructions controlling related hardware, or by dedicated circuits, programmable logic devices, or a combination thereof. Correspondingly, the systems, devices, modules, units, or components in the foregoing embodiments can be implemented in software, hardware, or a combination of both. The division of modules, units, or components is merely a logical division for the purpose of illustrating the technical solution; in actual implementation, they can be combined, split, or integrated as needed.

[0164] In one embodiment, the electronic device may include a processor, a memory, and a communication interface, wherein the memory is used to store program instructions, and the processor is used to call and execute the program instructions to implement all or part of the steps in the foregoing method embodiments. The electronic device may be a server, a terminal device, an edge computing node, a cloud computing device, or other device with data processing capabilities.

[0165] In one embodiment, this application may also be implemented in the form of a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to implement all or part of the steps in the foregoing method embodiments. The computer-readable storage medium may be a read-only memory, random access memory, flash memory, hard disk, solid-state drive, optical disk, or other non-transitory storage medium.

[0166] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, system embodiments are basically similar to method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. In the description of this specification, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0167] Furthermore, the terms "including," "comprising," and "having" used in the specification are all non-exclusive inclusions; the terms "first," "second," etc., are only used to distinguish technical features and do not indicate limitations on order, quantity, or importance. The execution order of each step in the method embodiments is also not absolutely limited. Without departing from the technical concept of this application, the steps can be adjusted in order, executed in parallel, combined, or split for execution.

Claims

1. A discrete action correction method for offline reinforcement learning in vision, characterized in that, include: Obtain an offline sample set and generate a discrete latent action representation based on the state transition information in the offline sample set; Based on the discrete latent action representation, an action evaluation result is generated, and based on the relationship between the action evaluation result and the original action labels in the offline sample set, the sample reliability result and the overall contamination level result are determined. The original action labels in the offline sample set are adjusted based on the sample reliability results and the overall contamination level results to obtain updated action labels; The offline sample set is corrected based on the updated action labels.

2. The discrete motion correction method according to claim 1, characterized in that, The step of generating discrete latent action representations based on state transition information in the offline sample set includes: The observation data and the next time-instance observation are obtained from the offline sample set as the state transition related information; A state transition representation is constructed based on the observation data and the observation at the next time step. The discrete latent action representation is generated based on the state transition representation.

3. The discrete motion correction method according to claim 2, characterized in that, The step of generating the discrete latent action representation based on the state transition representation includes: Generate a continuous latent action representation based on the state transition representation; The continuous latent action representation is discretized to obtain the discrete latent action representation; The step of generating an action evaluation result based on the discrete latent action representation includes: generating an action prediction result based on the discrete latent action representation, and using the action prediction result as the action evaluation result.

4. The discrete motion correction method according to claim 3, characterized in that, The step of determining the sample reliability result and the overall contamination level result based on the relationship between the action evaluation result and the original action labels in the offline sample set includes: The sample prediction loss is determined based on the deviation between the action prediction result, which is the result of the action evaluation, and the original action label. The sample reliability result and the overall contamination level result are determined based on the sample prediction loss.

5. The discrete motion correction method according to claim 4, characterized in that, The step of determining the sample reliability result and the overall contamination level result based on the sample prediction loss includes: The prediction loss of the sample is decomposed into a distribution. The sample cleanliness is determined as the sample reliability result based on the distribution decomposition results; The overall pollution level is determined based on the distribution decomposition results.

6. The discrete motion correction method according to claim 5, characterized in that, The step of adjusting the original action labels in the offline sample set based on the sample reliability results and the overall contamination level results to obtain updated action labels includes: The action update conditions are determined based on the sample reliability results and the overall contamination level results. When the action update condition is met, the original action label is replaced with the predicted action label corresponding to the action prediction result; If the action update condition is not met, the original action tag is retained; The updated action tags are obtained based on either the replaced action tags or the retained original action tags.

7. The discrete motion correction method according to claim 6, characterized in that, The step of correcting the offline sample set based on the updated action labels includes: The updated action labels are used to replace the original action labels in the offline sample set, while retaining the observation data, reward information, and next-time observation in the offline sample set, to generate a corrected offline dataset; Subsequent robust training is performed based on the corrected offline dataset to obtain the target policy.

8. A discrete action correction system for offline reinforcement learning in vision, characterized in that, include: The latent action generation module is used to acquire an offline sample set and generate discrete latent action representations based on state transition information in the offline sample set. The evaluation module is used to generate action evaluation results based on the discrete latent action representation, and to determine the sample reliability results and the overall contamination level results based on the relationship between the action evaluation results and the original action labels in the offline sample set. The label adjustment module is used to adjust the original action labels in the offline sample set according to the sample reliability results and the overall contamination results, so as to obtain updated action labels; The sample correction module is used to correct the offline sample set based on the updated action labels.

9. An electronic device, characterized in that, include: Memory; processor; And a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the discrete motion correction method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the discrete motion correction method according to any one of claims 1 to 7.

Images