Navigation model distributed training method and device, computer equipment and storage medium
Through the distributed training method, the agent cluster trains navigation models based on local data, solving the problem that navigation model training in the existing technology is not suitable for the environment and privacy data leakage, and achieving higher intelligence and privacy security.
Patent Information
- Application Number
- CN202411240539.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-04
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-09-04
Smart Images

Figure CN119066428A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of machine vision, and in particular to a navigation model distributed training method, apparatus, computer equipment and storage medium. Background Art
[0002] An intelligent agent refers to an entity with intelligence. Taking a household robot as an example, the household robot can use a network model to digitally describe the environmental information collected by machine vision and the information of objects contained in the environment, such as walls, furniture, etc., and then convert the digital description into corresponding action strategies based on the network model, so that the household robot can move in the environment according to the action strategies.
[0003] However, for the network models currently commonly used in intelligent navigation, their training process is often completed on a single server, which makes it impossible to conduct personalized model training based on the actual scene information of the environment in which each intelligent agent is located. When the trained model is deployed on each intelligent agent, problems such as incompatibility with the environment in which the intelligent agent is located or lack of intelligence may occur, such as inaccurate environmental perception and fixed navigation paths. In addition, if the local data of each intelligent agent is uploaded to the same server for training the network model, it may bring security issues such as leakage of local privacy data or sensitive data of the intelligent agent. Summary of the invention
[0004] The embodiments of the present application provide a navigation model distributed training method, apparatus, computer equipment and storage medium, which are intended to improve the intelligence of the trained navigation model, as well as the adaptability of the navigation model to the intelligent agent and the environment in which the intelligent agent is located when the navigation model is applied to the intelligent agent, and to avoid the leakage of the local privacy data or sensitive data of the intelligent agent during the training process.
[0005] In a first aspect, an embodiment of the present application provides a navigation model distributed training method, including:
[0006] Acquire an environment image library and call an initial encoding model, the environment image library including a plurality of environment images and natural language instructions corresponding to the environment images one by one;
[0007] The initial coding model is trained based on the environment image and the natural language instruction to obtain a target coding model, wherein the target coding model outputs a descriptive code representing the environment image, and the descriptive code matches the natural language instruction corresponding to the environment image;
[0008] generating a global navigation model based on the target encoding model, wherein the global navigation model is used to generate an action strategy according to an environment image;
[0009] The global navigation model is sent to each agent in the agent cluster respectively, so that each agent trains the model parameters in the global navigation model based on the corresponding local data to obtain the first training parameters;
[0010] The first training parameters fed back by each agent are obtained, and the parameters of the global navigation model are updated according to the first training parameters to generate a target navigation model.
[0011] In some implementations, updating the parameters of the global navigation model according to the first training parameter to generate a target navigation model includes:
[0012] Generate an aggregation parameter according to the first training parameter corresponding to each agent;
[0013] Replacing the model parameters in the global navigation model with the aggregated parameters to obtain the pending navigation model;
[0014] When the pending navigation model meets a preset convergence criterion, the pending navigation model is used as a target navigation model.
[0015] In some embodiments, the method further comprises:
[0016] When the pending navigation model does not meet the preset convergence criteria, the aggregated parameters are sent to each agent, so that the agent obtains a first navigation model by replacing the model parameters in the global navigation model with the aggregated parameters, and trains the model parameters in the first navigation model based on the corresponding local data to obtain second training parameters;
[0017] The second training parameters fed back by each agent are obtained, and the parameters of the global navigation model are updated according to the first training parameters to generate a revised navigation model.
[0018] In some implementations, generating the aggregate parameter according to the first training parameter corresponding to each agent includes:
[0019] Get the weight coefficient corresponding to each agent;
[0020] The weighted average is calculated according to the first training parameter corresponding to the agent and the weight coefficient as the aggregation parameter.
[0021] In some implementations, generating a global navigation model based on a target coding model includes:
[0022] Retrieving a first linear network, the first linear network is used to output an action strategy generated based on the descriptive code;
[0023] Generate a strategy model according to the target encoding model and the first linear network;
[0024] Retrieving a second linear network, the second linear network is used to output an action value determined by evaluating the action strategy based on the descriptive code;
[0025] Generate a value model according to the target encoding model and the second linear network;
[0026] Generate a global navigation model based on the strategy model and value model.
[0027] In some implementations, before training the initial encoding model based on the environment image and the natural language instruction to obtain the target encoding model, the method further includes:
[0028] Performing an image size adjustment operation on the environment image so that the size of the adjusted environment image meets a preset size range;
[0029] and / or,
[0030] A pixel value adjustment operation is performed on at least part of the pixels in the environment image, where the pixel value adjustment operation includes at least one of normalization, sharpening, and contrast adjustment.
[0031] In some implementations, training an initial encoding model based on an environment image and a natural language instruction to obtain a target encoding model includes:
[0032] The environment image is input into the initial coding model, and the initial coding model outputs a descriptive code corresponding to the environment image;
[0033] Performing encoding operations on the natural language instructions to generate language instruction encodings, and generating semantic feature vectors representing the similarity between the descriptive encodings and the language instruction encodings;
[0034] The model parameters in the initial coding model are iteratively trained based on the semantic feature vector to obtain the target coding model.
[0035] In a second aspect, an embodiment of the present application further provides a navigation model distributed training device, including:
[0036] An initialization module, used to obtain an environment image library and call an initial coding model, wherein the environment image library includes a plurality of environment images and natural language instructions corresponding to the environment images one by one;
[0037] A centralized training module is used to train the initial coding model based on the environment image and the natural language instruction to obtain a target coding model, wherein the target coding model outputs a descriptive code representing the environment image, and the descriptive code matches the natural language instruction corresponding to the environment image;
[0038] A global model module, used for generating a global navigation model based on the target encoding model, wherein the global navigation model is used for generating an action strategy according to an environment image;
[0039] A model sending module, used for sending the global navigation model to each agent in the agent cluster respectively, so that each agent trains the model parameters in the global navigation model based on the corresponding local data to obtain the first training parameters;
[0040] The model aggregation module is used to obtain the first training parameters fed back by each intelligent agent, and update the parameters of the global navigation model according to the first training parameters to generate a target navigation model.
[0041] In a third aspect, an embodiment of the present application further provides a computer device, including a memory and a processor;
[0042] Memory for storing computer programs;
[0043] A processor is used to execute a computer program and implement the above-mentioned navigation model distributed training method when executing the computer program.
[0044] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the above-mentioned navigation model distributed training method.
[0045] The embodiment of the present application provides a navigation model distributed training method, device, computer equipment and storage medium, wherein the navigation model distributed training method includes: obtaining an environment image library and calling an initial coding model, the environment image library includes a number of environment images and natural language instructions corresponding to the environment images; training the initial coding model based on the environment images and natural language instructions to obtain a target coding model, wherein the target coding model outputs a descriptive code representing the environment image, and the descriptive code matches the natural language instructions corresponding to the environment image; generating a global navigation model based on the target coding model, wherein the global navigation model is used to generate an action strategy according to the environment image; sending the global navigation model to each agent in the agent cluster respectively, so that each agent trains the model parameters in the global navigation model based on the corresponding local data to obtain a first training parameter; obtaining the first training parameter fed back by each agent, and updating the parameters of the global navigation model according to the first training parameter to generate a target navigation model. The embodiment of the present application deploys the global navigation model to multiple distributed agents, allowing each agent to train based on local data, without centralizing the local data of the agent to a central node, avoiding centralized storage and transmission of data, thereby improving the privacy security of local data of each agent. Moreover, deploying the global navigation model to each agent can better combine the agent's own environment and usage scenarios for training, so that the agent can learn more localized features and scenarios and better adapt to different environments and task requirements. In addition, since each agent trains the global navigation model issued based on local data, the update of the global model parameters is performed independently in different agents, making the model update process relatively easy to monitor and debug. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0047] Figure 1 A schematic diagram of the steps of a distributed training method for a navigation model provided in an embodiment of the present application;
[0048] Figure 2 A schematic diagram of iterative parameter training of an initial coding model in a navigation model distributed training method provided in an embodiment of the present application;
[0049] Figure 3 A schematic diagram of the model structure of a global navigation model in a navigation model distributed training method provided in an embodiment of the present application;
[0050] Figure 4 A schematic diagram of a management device and an agent cluster cooperating to perform model training in a navigation model distributed training method provided in an embodiment of the present application;
[0051] Figure 5 A schematic diagram of a process flow of generating a target navigation model in a navigation model distributed training method provided in an embodiment of the present application;
[0052] Figure 6 A schematic diagram of a process flow of generating a target navigation model in a navigation model distributed training method provided in an embodiment of the present application;
[0053] Figure 7 A schematic diagram of the module structure of a navigation model distributed training device provided in an embodiment of the present application;
[0054] Figure 8 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0055] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0056] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.
[0057] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0058] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0059] The embodiment of the present application provides a navigation model distributed training method, apparatus, computer equipment and storage medium, based on which the navigation model distributed training method is used to train a target navigation model applicable to agent navigation. The navigation model distributed training method can be applied to a management device connected to an agent cluster or at least one agent in the agent cluster, and it should be noted that the agent cluster includes at least two agents in a distributed setting.
[0060] The management device and the agent can be electronic devices such as computers, intelligent robots, independent servers or server clusters, which are not limited here. In this embodiment, the navigation model distributed training method is applied to the management device connected to the agent cluster communication, and the agent is a household robot as an example for explanation, but the navigation model distributed training method is not limited to being applied only to the management device connected to the agent cluster communication.
[0061] Some implementation modes of the present application are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments may be combined with each other.
[0062] See also Figure 1 , Figure 1 A schematic flowchart of the steps of a navigation model distributed training method provided in an embodiment of the present application.
[0063] like Figure 1 As shown, the method at least includes the following steps S1-S5.
[0064] Step S1, obtaining an environment image library and calling an initial coding model, wherein the environment image library includes a plurality of environment images and natural language instructions corresponding to the environment images one by one.
[0065] Specifically, the initial coding model is used to generate and output corresponding descriptive codes based on the input sample image conversion. Thus, the descriptive codes generated by the sample image conversion can be compared with the natural language instructions corresponding to the sample image to determine the degree of match between the descriptive codes and the natural language instructions.
[0066] It should be noted that the sample images include environment images, and the environment images in the environment image library form a one-to-one correspondence with the natural language instructions. When the environment images in the environment image library are input into the initial coding model, the initial coding model generates a descriptive code corresponding to the environment image based on the environment image.
[0067] It should also be noted that the natural language instruction is a natural language text of any length, and its text content is a pre-annotated instruction text for the environment image. For example, the text content of the natural language instruction describes the instructions made by the household robot when facing various environments and external objects, and the multiple environment images in the environment image library cover as many environments and external objects as possible that the household robot may face.
[0068] Step S2: training the initial coding model based on the environment image and the natural language instructions to obtain a target coding model, wherein the target coding model outputs a descriptive code representing the environment image, and the descriptive code matches the natural language instructions corresponding to the environment image.
[0069] After obtaining the initial coding model and the environmental image library, the initial coding model is trained based on the environmental images in the environmental image library and the natural language instructions corresponding to the environmental images. Specifically, when the environmental image is input into the initial coding model, the initial coding model generates a descriptive code corresponding to the environmental image based on the environmental image, and then compares the descriptive code generated by the conversion with the corresponding natural language instructions, so as to adjust the parameters in the initial coding model according to the comparison results.
[0070] It should be understood that in the comparison result, the greater the difference between the descriptive code and the natural language instruction, the lower the accuracy of the descriptive code generated by the initial coding model, and vice versa, the smaller the difference between the descriptive code and the natural language instruction, the higher the accuracy of the descriptive code generated by the initial coding model, and the goal of training the initial coding model in this embodiment is to train the target coding model to generate accurate descriptive codes according to the environment image. Therefore, the convergence condition for training the initial coding model includes at least the matching between the natural language instruction corresponding to the environment image and the descriptive code.
[0071] In some implementations, training an initial encoding model based on an environment image and a natural language instruction to obtain a target encoding model includes:
[0072] The environment image is input into the initial coding model, and the initial coding model outputs a descriptive code corresponding to the environment image;
[0073] Performing encoding operations on the natural language instructions to generate language instruction encodings, and generating semantic feature vectors representing the similarity between the descriptive encodings and the language instruction encodings;
[0074] The model parameters in the initial coding model are iteratively trained based on the semantic feature vector to obtain the target coding model.
[0075] See also Figure 2 , Figure 2A schematic diagram of iterative parameter training of an initial coding model in a navigation model distributed training method provided in an embodiment of the present application.
[0076] like Figure 2 As shown, the iterative parameter training of the initial coding model is specifically explained: first, the order of each environmental image is determined or a separate sequence number is assigned to each environmental image, and then the environmental image 1 is input into the initial coding model 0, and the initial coding model 0 outputs the descriptive code 1 corresponding to the environmental image 1; at the same time, the natural language instruction 1 corresponding to the environmental image 1 is encoded to generate a language instruction code 1, and a semantic feature vector 1 representing the similarity between the descriptive code 1 corresponding to the environmental image 1 and the language instruction code 1 is generated, and the parameters in the current initial coding model 0 are adjusted based on the semantic feature vector 1 to obtain the initial coding model 1, and the first training is completed; then, the environmental image 2 is input into the initial coding model 1 obtained after completing a parameter training, and the second training is completed in the same way as the first training, and so on. After completing several trainings, the corresponding semantic feature vector meets the preset first convergence condition, and the iterative training is considered to be completed.
[0077] For example, the semantic feature vector represents the cosine similarity between a descriptive code and a language instruction code, and the training goal in the embodiment of the present application is to maximize the cosine similarity between the descriptive code and the language instruction code corresponding to the same environmental image, and minimize the cosine similarity between the descriptive code and the language instruction code corresponding to different environmental images.
[0078] Exemplarily, the convergence condition for training the initial coding model is: the cosine similarity between the descriptive coding and the language instruction coding corresponding to the same environmental image is greater than a first preset value, and / or the cosine similarity between the descriptive coding and the language instruction coding corresponding to different environmental images is less than a second preset value.
[0079] Exemplarily, in this embodiment, some network layers in a CLIP (Contrast i ve Language-Image Pre-training) model can be applied as an initial coding model, and training is performed based on it to obtain a target coding model.
[0080] Furthermore, the CLIP model can be used to complete the encoding operation of the natural language instruction to generate the language instruction code, and to generate the semantic feature vector representing the similarity between the descriptive code and the language instruction code.
[0081] In some implementations, before training the initial encoding model based on the environment image and the natural language instruction to obtain the target encoding model, the method further includes:
[0082] Performing an image size adjustment operation on the environment image so that the size of the adjusted environment image meets a preset size range;
[0083] and / or,
[0084] A pixel value adjustment operation is performed on at least part of the pixels in the environment image, where the pixel value adjustment operation includes at least one of normalization, sharpening, and contrast adjustment.
[0085] It should be understood that before training the initial coding model, performing image resizing operations and pixel value adjustment operations on the environment image can effectively reduce the image size of the environment image, thereby reducing the amount of subsequent calculations on the initial coding model and speeding up iterative training.
[0086] Step S3: Generate a global navigation model based on the target coding model, wherein the global navigation model is used to generate an action strategy according to the environment image.
[0087] Exemplarily, the global navigation model is used to generate descriptive codes according to the environment image, and then generate an action strategy for the scene based on the descriptive code of the environment image, and generate an action value for the action strategy based on the environment image to judge the quality of the action strategy.
[0088] Specifically, the global navigation model is composed of a strategy model and a value model, and the strategy model is used to generate a descriptive code according to the environment image, and then generate an action strategy for the scene based on the descriptive code of the environment image. The value model is used to generate an action value for judging the quality of the action strategy based on the action strategy output by the strategy model based on the environment image. The following is an implementation method for generating a global navigation model based on the target coding model obtained by training and the combination of the called first linear network and the second linear network:
[0089] See also Figure 3 , Figure 3 A schematic diagram of the model structure of a global navigation model in a navigation model distributed training method provided in an embodiment of the present application. Figure 3 (a) is a schematic diagram of a target coding model, a first linear network and a second linear network, Figure 3 (b) is a schematic diagram of the strategy model and value model.
[0090] like Figure 3 As shown, in some implementations, generating a global navigation model based on a target coding model includes:
[0091] Retrieving a first linear network, the first linear network is used to output an action strategy generated based on the descriptive code;
[0092] Generate a strategy model according to the target encoding model and the first linear network;
[0093] Retrieving a second linear network, the second linear network is used to output an action value determined by evaluating the action strategy based on the descriptive code;
[0094] Generate a value model according to the target encoding model and the second linear network;
[0095] Generate a global navigation model based on the strategy model and value model.
[0096] Specifically, the first linear network is used to generate an action strategy for the scene based on the input descriptive code. Exemplarily, the first linear network is a preset Actor network.
[0097] Specifically, the second linear network is used to generate an action value for judging the quality of the action strategy based on the environment image output by the first linear network. Exemplarily, the second linear network is a preset critical network.
[0098] Therefore, combining the target coding model with the first linear network can obtain the policy model in the global navigation model, combining the target coding model with the second linear network can obtain the value model in the global navigation model, and then combining the policy model with the value model to obtain the global navigation model.
[0099] Step S4: Send the global navigation model to each agent in the agent cluster respectively, so that each agent trains the model parameters in the global navigation model based on the corresponding local data to obtain the first training parameters.
[0100] See also Figure 4 , Figure 4 A schematic diagram of a management device and an agent cluster cooperating to perform model training in a navigation model distributed training method provided in an embodiment of the present application.
[0101] like Figure 3 and 4 As shown, the distributed training method of the navigation model is applied to a management device connected to a cluster of intelligent agents, and the intelligent agent cluster is used as an example for specific description:
[0102] Specifically, the management device transmits the trained global navigation model to the network through wired communication or wireless communication (e.g. Figure 4The example is distributed to each agent in the agent cluster in the form of management device broadcast), so that each agent performs parameter training on the strategy model and the value model in the global navigation model based on the corresponding local data until the global navigation model converges, and obtains the first navigation model corresponding to the agent, and the first training parameters are the model parameters of the first navigation model. Therefore, the first training parameters are obtained by the agent training on the model parameters in the global navigation model.
[0103] It should be noted that the convergence conditions of the global navigation model are related to the training methods adopted by each intelligent agent. The local data includes local environment images and labeled language instructions corresponding to the local environment images. The labeled language instructions are natural language texts of arbitrarily length, and their text content is prescriptive text pre-annotated for the local environment images. The intelligent agent inputs the local environment images into the global navigation model. The strategy model in the global navigation model generates action strategies based on the local environment images, and the value model generates action values based on the action strategies. Then, the action network and the value model are trained according to the annotations and action values attached to the local environment images until the action values of the action strategies made by the global navigation model for different local environment images can stably reach the target values.
[0104] Exemplarily, the process in which the agent locally trains parameters of the strategy model and the value model in the global navigation model can specifically adopt the DD-PPO (Decentralized Distributed Proximal Policy Optimization) algorithm.
[0105] Step S5, obtaining training parameters fed back by the agent cluster, and updating the parameters of the global navigation model according to the training parameters to generate a target navigation model, wherein the training parameters include model parameters of the strategy model and the value model in the first navigation model corresponding to each agent.
[0106] See also Figure 5 , Figure 5 A schematic flow chart of the steps for generating a target navigation model in a navigation model distributed training method provided in an embodiment of the present application.
[0107] like Figure 5 As shown, in some embodiments, obtaining the first training parameters fed back by each agent, and updating the parameters of the global navigation model according to the first training parameters to generate a target navigation model includes steps S51 to S53:
[0108] Step S51, generating aggregation parameters according to the first training parameters corresponding to each agent;
[0109] Step S52: using the aggregated parameters to replace the model parameters in the global navigation model to obtain a pending navigation model;
[0110] Step S53: When the pending navigation model meets a preset convergence criterion, the pending navigation model is used as a target navigation model.
[0111] Specifically, the first training parameter refers to the model parameters of the strategy model and the value model obtained by each intelligent agent in the process of training the global navigation model locally. The management device that executes this method aggregates parameters according to the first training parameters corresponding to each intelligent agent to obtain aggregate parameters, and then uses the aggregate parameters to replace the parameters of the strategy model and the value model in the global navigation model originally stored in the management device to obtain a pending navigation model. Then, it is determined whether the pending navigation model meets the preset iterative convergence rules. If the pending navigation model meets the iterative convergence rules, the pending navigation model is used as the target navigation model and sent to each intelligent agent.
[0112] After the target navigation model is sent to the intelligent agent, the intelligent agent can deploy the target navigation model and apply the target navigation model to perform navigation operations after the deployment is completed. The intelligent agent can use the target navigation model to convert environmental information collected through machine vision into corresponding action strategies, thereby achieving effective navigation.
[0113] In some implementations, generating the aggregation parameter according to the first training parameter corresponding to each agent in step S51 includes:
[0114] Get the weight coefficient corresponding to each agent;
[0115] A weighted average is calculated according to the first training parameter corresponding to the agent and the weight coefficient as the aggregation parameter.
[0116] Specifically, the aggregation parameters include the first model parameters of the strategy model in the first navigation model and the second model parameters in the strategy model. The weighted average value is calculated according to the first training parameters and weight coefficients corresponding to the intelligent agent as the aggregation parameter, including: calculating the weighted average value according to the model parameters and weight coefficients of the strategy model fed back by each intelligent agent as the first model parameter; and / or calculating the weighted average value according to the model parameters and weight coefficients of the value model fed back by each intelligent agent as the second model parameter.
[0117] It should be understood that in an agent cluster, the sum of the weight coefficients corresponding to each agent should be 1.
[0118] It should be understood that by calculating the weighted average of the policy model parameters corresponding to each agent as the first model coefficient / second model coefficient, the local training results of different agents can be fully combined to generate the aggregation parameters to be applied to the navigation model to be determined. Furthermore, the weight coefficients corresponding to each agent can be adjusted, and the degree of influence of the local training results of each agent on the aggregation parameters can be effectively controlled by adjusting the weight parameters.
[0119] See also Figure 6 , Figure 6 A schematic flow chart of the steps for generating a target navigation model in a navigation model distributed training method provided in an embodiment of the present application.
[0120] like Figure 6 As shown, in some embodiments, the method further includes steps S54 to S55:
[0121] Step S54: when the pending navigation model does not meet the preset convergence criteria, the aggregated parameters are sent to each agent, so that the agent obtains a first navigation model by replacing the model parameters in the global navigation model with the aggregated parameters, and trains the model parameters in the first navigation model based on the corresponding local data to obtain second training parameters;
[0122] Step S55: obtaining the second training parameters fed back by each agent, and updating the parameters of the global navigation model according to the first training parameters to generate a revised navigation model.
[0123] Specifically, when the pending navigation model does not meet the preset convergence criteria, the aggregation parameters are sent to each intelligent agent in the intelligent agent cluster, so that the intelligent agent obtains the first navigation model by replacing the network parameters of the strategy model and the value model in the global navigation model with the aggregation parameters, and trains the model parameters in the first navigation model based on the corresponding local data to obtain the second training parameters.
[0124] After this, the management device executing the present method obtains the second training parameters fed back by each intelligent agent, and uses the second training parameters to replace the parameters of the strategy model and the value model in the pending navigation model to obtain a revised navigation model, and determines whether the revised navigation model after the parameter replacement meets the preset convergence criteria. If the revised navigation model after the parameter replacement meets the preset iterative convergence rules at this time, the pending navigation model is used as the target navigation model and sent to the intelligent agent cluster. If the revised navigation model after the parameter replacement still does not meet the preset convergence criteria, the revised navigation model is used as the pending navigation model, and steps S54 and S55 are repeated until the obtained revised navigation model meets the preset iterative convergence rules.
[0125] In summary, the embodiments of the present application deploy the global navigation model to multiple distributed agents, allowing each agent to be trained based on local data, without centralizing the local data of the agent to a central node, avoiding centralized storage and transmission of data, and thus improving the privacy security of each agent's local data. Moreover, deploying the global navigation model to each agent can better combine the environment and usage scenarios of the agent itself for training, so that the agent can learn more localized features and scenarios, and better adapt to different environments and task requirements. In addition, since each agent is trained on the global navigation model issued based on local data, the update of the global model parameters is performed independently in different agents, making the model update process relatively easy to monitor and debug.
[0126] See also Figure 7 , Figure 7 A schematic diagram of the module structure of a navigation model distributed training device provided in an embodiment of the present application.
[0127] like Figure 7 As shown, the navigation model distributed training device 600 includes:
[0128] Initialization module 601, used to obtain an environment image library and call an initial coding model, the environment image library includes a number of environment images and natural language instructions corresponding to the environment images one by one;
[0129] A centralized training module 602 is used to train the initial coding model based on the environment image and the natural language instruction to obtain a target coding model, wherein the target coding model outputs a descriptive code representing the environment image, and the descriptive code matches the natural language instruction corresponding to the environment image;
[0130] A global model module 603, used to generate a global navigation model based on the target coding model, wherein the global navigation model is used to generate an action strategy according to the environment image;
[0131] The model sending module 604 is used to send the global navigation model to each agent in the agent cluster, so that each agent trains the model parameters in the global navigation model based on the corresponding local data to obtain the first training parameters;
[0132] The model aggregation module 605 is used to obtain the first training parameters fed back by each agent, and update the parameters of the global navigation model according to the first training parameters to generate a target navigation model.
[0133] In some implementations, the model aggregation module 605 updates the parameters of the global navigation model according to the first training parameters to generate a target navigation model, including:
[0134] Generate aggregation parameters according to the first training parameters corresponding to each agent;
[0135] Replacing the model parameters in the global navigation model with the aggregated parameters to obtain the pending navigation model;
[0136] When the pending navigation model meets a preset convergence criterion, the pending navigation model is used as a target navigation model.
[0137] In some implementations, the model aggregation module 605 updates the parameters of the global navigation model according to the first training parameters to generate a target navigation model, and further includes:
[0138] When the pending navigation model does not meet the preset convergence criteria, the aggregated parameters are sent to each agent, so that the agent obtains a first navigation model by replacing the model parameters in the global navigation model with the aggregated parameters, and trains the model parameters in the first navigation model based on the corresponding local data to obtain second training parameters;
[0139] The second training parameters fed back by each agent are obtained, and the parameters of the global navigation model are updated according to the first training parameters to generate a revised navigation model.
[0140] In some implementations, the model aggregation module 605 generates aggregation parameters according to the first training parameters corresponding to each agent, including:
[0141] Get the weight coefficient corresponding to each agent;
[0142] A weighted average is calculated according to the first training parameter corresponding to the agent and the weight coefficient as the aggregation parameter.
[0143] In some implementations, the global model module 603 generates a global navigation model based on the target coding model, including:
[0144] Retrieving a first linear network, the first linear network is used to output an action strategy generated based on the descriptive code;
[0145] Generate a strategy model according to the target encoding model and the first linear network;
[0146] Retrieving a second linear network, the second linear network is used to output an action value determined by evaluating the action strategy based on the descriptive code;
[0147] Generate a value model according to the target encoding model and the second linear network;
[0148] Generate a global navigation model based on the strategy model and value model.
[0149] In some implementations, before the centralized training module 602 trains the initial coding model based on the environment image and the natural language instruction to obtain the target coding model, the centralized training module 602 further includes:
[0150] Performing an image size adjustment operation on the environment image so that the size of the adjusted environment image meets a preset size range;
[0151] and / or,
[0152] A pixel value adjustment operation is performed on at least part of the pixels in the environment image, where the pixel value adjustment operation includes at least one of normalization, sharpening, and contrast adjustment.
[0153] In some implementations, the centralized training module 602 trains the initial coding model based on the environment image and the natural language instruction to obtain the target coding model, including:
[0154] The environment image is input into the initial coding model, and the initial coding model outputs a descriptive code corresponding to the environment image;
[0155] Performing encoding operations on the natural language instructions to generate language instruction encodings, and generating semantic feature vectors representing the similarity between the descriptive encodings and the language instruction encodings;
[0156] The model parameters in the initial coding model are iteratively trained based on the semantic feature vector to obtain the target coding model.
[0157] See also Figure 8 , Figure 8 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application.
[0158] like Figure 8 As shown, the computer device 700 includes a processor 701 and a memory 702, and the processor 701 and the memory 702 are connected via a bus 703, such as an I2C (Inter-Integrated Circuit) bus.
[0159] Specifically, the processor 701 is used to provide computing and control capabilities to support the operation of the entire computer device. The processor 701 can be a central processing unit (CPU), and the processor 701 can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.
[0160] Specifically, the memory 702 can be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB disk, or a mobile hard disk.
[0161] Those skilled in the art will understand that Figure 8 The structure shown in the figure is merely a block diagram of a partial structure related to the embodiment of the present application, and does not constitute a limitation on the computer device to which the embodiment of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0162] The processor is used to run a computer program stored in the memory, and implement any one of the navigation model distributed training methods provided in the embodiments of the present application when executing the computer program.
[0163] In one embodiment, the processor 701 is used to run a computer program stored in the memory 702, and implements the following steps when executing the computer program:
[0164] Acquire an environment image library and call an initial encoding model, the environment image library including a plurality of environment images and natural language instructions corresponding to the environment images one by one;
[0165] The initial coding model is trained based on the environment image and the natural language instruction to obtain a target coding model, wherein the target coding model outputs a descriptive code representing the environment image, and the descriptive code matches the natural language instruction corresponding to the environment image;
[0166] generating a global navigation model based on the target encoding model, wherein the global navigation model is used to generate an action strategy according to an environment image;
[0167] The global navigation model is sent to each agent in the agent cluster respectively, so that each agent trains the model parameters in the global navigation model based on the corresponding local data to obtain the first training parameters;
[0168] The first training parameters fed back by each agent are obtained, and the parameters of the global navigation model are updated according to the first training parameters to generate a target navigation model.
[0169] In some implementations, when the processor 701 updates the parameters of the global navigation model according to the first training parameter to generate a target navigation model, it includes:
[0170] Generate an aggregation parameter according to the first training parameter corresponding to each agent;
[0171] Replacing the model parameters in the global navigation model with the aggregated parameters to obtain the pending navigation model;
[0172] When the pending navigation model meets a preset convergence criterion, the pending navigation model is used as a target navigation model.
[0173] In some implementations, when the processor 701 updates the parameters of the global navigation model according to the first training parameter to generate a target navigation model, the process further includes:
[0174] When the pending navigation model does not meet the preset convergence criteria, the aggregated parameters are sent to each agent, so that the agent obtains a first navigation model by replacing the model parameters in the global navigation model with the aggregated parameters, and trains the model parameters in the first navigation model based on the corresponding local data to obtain second training parameters;
[0175] The second training parameters fed back by each agent are obtained, and the parameters of the global navigation model are updated according to the first training parameters to generate a revised navigation model.
[0176] In some implementations, when the processor 701 generates the aggregate parameter according to the first training parameter corresponding to each agent, it includes:
[0177] Get the weight coefficient corresponding to each agent;
[0178] A weighted average is calculated according to the first training parameter corresponding to the agent and the weight coefficient as the aggregation parameter.
[0179] In some implementations, when the processor 701 generates a global navigation model based on the target coding model, the process includes:
[0180] Retrieving a first linear network, the first linear network is used to output an action strategy generated based on the descriptive code;
[0181] Generate a strategy model according to the target encoding model and the first linear network;
[0182] Retrieving a second linear network, the second linear network is used to output an action value determined by evaluating the action strategy based on the descriptive code;
[0183] Generate a value model according to the target encoding model and the second linear network;
[0184] Generate a global navigation model based on the strategy model and value model.
[0185] In some implementations, before the processor 701 trains the initial coding model based on the environment image and the natural language instruction to obtain the target coding model, the process further includes:
[0186] Performing an image size adjustment operation on the environment image so that the size of the adjusted environment image meets a preset size range;
[0187] and / or,
[0188] A pixel value adjustment operation is performed on at least part of the pixels in the environment image, where the pixel value adjustment operation includes at least one of normalization, sharpening, and contrast adjustment.
[0189] In some implementations, when the processor 701 trains the initial coding model based on the environment image and the natural language instruction to obtain the target coding model, the process includes:
[0190] The environment image is input into the initial coding model, and the initial coding model outputs a descriptive code corresponding to the environment image;
[0191] Performing encoding operations on the natural language instructions to generate language instruction encodings, and generating semantic feature vectors representing the similarity between the descriptive encodings and the language instruction encodings;
[0192] The model parameters in the initial coding model are iteratively trained based on the semantic feature vector to obtain the target coding model.
[0193] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the computer device described above can refer to the corresponding process in the aforementioned animal identification method embodiment, and will not be repeated here.
[0194] An embodiment of the present application also provides a storage medium storing a computer program, which can be executed by one or more processors to implement the steps of any navigation model distributed training method provided in the description of the embodiment of the present application.
[0195] The storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (Smart Med ia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.
[0196] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware embodiment, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.
[0197] It should be understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.
[0198] The serial numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments. The above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A navigation model distributed training method, characterized in that: The method comprises: Acquire an environment image library and call an initial coding model, wherein the environment image library includes a plurality of environment images and natural language instructions corresponding to the environment images one by one; Training the initial coding model based on the environment image and the natural language instruction to obtain a target coding model, wherein the target coding model outputs a descriptive code representing the environment image, and the descriptive code matches the natural language instruction corresponding to the environment image; generating a global navigation model based on the target coding model, wherein the global navigation model is used to generate an action strategy according to the environment image; The global navigation model is sent to each agent in the agent cluster respectively, so that each agent trains the model parameters in the global navigation model based on the corresponding local data to obtain the first training parameters; The first training parameters fed back by each of the agents are obtained, and parameters of the global navigation model are updated according to the first training parameters to generate a target navigation model.
2. The method according to claim 1, characterized in that The updating the parameters of the global navigation model according to the first training parameters to generate a target navigation model includes: Generate an aggregation parameter according to the first training parameter corresponding to each of the intelligent agents; Replacing the model parameters in the global navigation model with the aggregation parameters to obtain a pending navigation model; When the pending navigation model meets a preset convergence criterion, the pending navigation model is used as the target navigation model.
3. The method according to claim 2, characterized in that The method further comprises: When the pending navigation model does not meet the preset convergence criterion, the aggregation parameter is sent to each of the agents, so that the agents obtain a first navigation model by replacing the model parameters in the global navigation model with the aggregation parameter, and train the model parameters in the first navigation model based on the corresponding local data to obtain second training parameters; The second training parameters fed back by each of the agents are obtained, and the parameters of the global navigation model are updated according to the first training parameters to generate a revised navigation model.
4. The method according to claim 2, characterized in that: The generating of the aggregation parameter according to the first training parameter corresponding to each of the intelligent agents comprises: Obtaining weight coefficients corresponding to each of the intelligent agents; A weighted average is calculated according to the first training parameter corresponding to the agent and the weight coefficient as the aggregation parameter.
5. The method according to claim 1, characterized in that: The generating a global navigation model based on the target coding model comprises: Retrieving a first linear network, wherein the first linear network is used to output an action strategy generated based on the descriptive code; Generate a strategy model according to the target coding model and the first linear network; Retrieving a second linear network, the second linear network being used to output an action value determined by evaluating the action strategy based on the descriptive code; Generate a value model according to the target encoding model and the second linear network; The global navigation model is generated based on the policy model and the value model.
6. The method according to claim 1, characterized in that Before the initial coding model is trained based on the environment image and the natural language instruction to obtain the target coding model, the method further includes: Performing an image size adjustment operation on the environment image so that the size of the adjusted environment image meets a preset size range; and / or, A pixel value adjustment operation is performed on at least part of the pixels in the environment image, wherein the pixel value adjustment operation includes at least one of normalization, sharpening, and contrast adjustment.
7. The method according to any one of claims 1 to 6, characterized in that: The training of the initial coding model based on the environment image and the natural language instruction to obtain a target coding model includes: Inputting the environment image into the initial coding model, and having the initial coding model output a descriptive code corresponding to the environment image; Performing an encoding operation on the natural language instruction to generate a language instruction code, and generating a semantic feature vector representing the similarity between the descriptive code and the language instruction code; The model parameters in the initial coding model are iteratively trained based on the semantic feature vector to obtain a target coding model.
8. A navigation model distributed training device, characterized in that: include: An initialization module, used to obtain an environment image library and call an initial coding model, wherein the environment image library includes a plurality of environment images and natural language instructions corresponding to the environment images one by one; A centralized training module, configured to train the initial coding model based on the environment image and the natural language instruction to obtain a target coding model, wherein the target coding model outputs a descriptive code representing the environment image, and the descriptive code matches the natural language instruction corresponding to the environment image; A global model module, used for generating a global navigation model based on the target coding model, wherein the global navigation model is used for generating an action strategy according to the environment image; A model sending module, used for sending the global navigation model to each agent in the agent cluster respectively, so that each agent trains the model parameters in the global navigation model based on the corresponding local data to obtain the first training parameters; The model aggregation module is used to obtain the first training parameters fed back by each of the intelligent agents, and update the parameters of the global navigation model according to the first training parameters to generate a target navigation model.
9. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program and implement the navigation model distributed training method according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the navigation model distributed training method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Federal learning-based model training method and system, and storage medium
CN115018019A
Training method and apparatus
EP4276691A1
