Visual language alignment enhancement method and device, equipment and medium
By constructing a sample set of image-text pairs and conducting comparative learning and training on the initial visual language model, domain adaptation and multimodal alignment optimization are performed for elderly care scenarios. This solves the problem of insufficient recognition ability of the visual language model in elderly care scenarios and improves the accuracy of target recognition and the naturalness of interaction.
Patent Information
- Application Number
- CN202510727439.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-30
- Publication Date
- 2025-09-16
Smart Images
Figure CN120656172A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology and can be applied to the field of medical health. In particular, it relates to a visual language alignment enhancement method, device, computer equipment and storage medium. Background Art
[0002] With the rapid development of artificial intelligence (AI) technology, indoor elderly care assistance robots are gradually demonstrating their practical application value in the elderly care sector, particularly in supporting the lives of elderly people with cognitive decline. These include delivering medications, finding hearing aids, and locating commonly used items. However, many of these robots rely on vision-language models (VLMs) to perform object recognition and understand natural language commands.
[0003] However, mainstream VLM models such as CLIP and BLIP rely heavily on training data from open internet image-text pairs (e.g., image-caption pairs), primarily covering scenarios such as daily life, commodities, e-commerce, and social media. This data suffers from domain bias, meaning there's a significant domain gap between the training distribution and actual elderly care scenarios. For example, items like specialized medication packaging, walkers, blood pressure monitors, and custom water cups used by the elderly are rarely found in public training data, leading to poor model performance in these scenarios.
[0004] Furthermore, the language descriptions used in elderly care scenarios exhibit significant semantic ambiguity, such as "that little red medicine bottle" and "the hearing aid I often use." These natural language expressions require semantic alignment with the long-tail visual objects in the environment. However, general VLM models are prone to failure in such long-tail, small-sample, and weakly semantically significant object recognition tasks, manifesting as object recognition errors, unresponsive searches, and command misunderstandings, reducing the reliability of human-computer interaction and user satisfaction.
[0005] Therefore, existing visual language models have insufficient recognition capabilities in elderly care scenarios. Summary of the Invention
[0006] The embodiments of the present invention provide a visual language alignment enhancement method, apparatus, computer device, and storage medium, aiming to solve the problem that the visual language model in the prior art has insufficient recognition ability in elderly care scenarios.
[0007] In a first aspect, an embodiment of the present invention provides a visual language alignment enhancement method, which includes:
[0008] Acquire an object image sample set and a text sample set, and construct an image-text pair sample set based on the object image sample set and the text sample set;
[0009] Add the adaptation module and projection head to the visual encoder and language encoder in the pre-trained visual language model to construct an initial visual language model;
[0010] Performing comparative learning training on the initial visual language model based on the image-text pair sample set and a preset loss calculation strategy to obtain a target visual language model;
[0011] If a user voice command is received, converting the user voice command into text data;
[0012] A target object image corresponding to the text data is obtained based on a current environment image collected in real time, the text data and the target visual language model.
[0013] In a second aspect, an embodiment of the present invention provides a visual language alignment enhancement device, comprising:
[0014] A sample construction unit, configured to obtain an object image sample set and a text sample set, and construct an image-text pair sample set based on the object image sample set and the text sample set;
[0015] A model construction unit is used to add the adaptation module and projection head to the visual encoder and language encoder in the pre-trained visual language model to construct an initial visual language model;
[0016] a model training unit, configured to perform comparative learning training on the initial visual language model based on the image-text pair sample set and a preset loss calculation strategy to obtain a target visual language model;
[0017] A voice processing unit, configured to convert a user voice instruction into text data upon receiving the user voice instruction;
[0018] The target recognition unit is used to obtain a target object image corresponding to the text data based on the current environment image collected in real time, the text data and the target visual language model.
[0019] In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the visual language alignment enhancement method of the first aspect described above is implemented.
[0020] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the visual language alignment enhancement method of the first aspect.
[0021] The embodiment of the present invention provides a visual language alignment enhancement method, device, equipment and medium, which includes constructing a picture-text pair sample set based on an object image sample set and a text sample set; adding an adaptation module and a projection head to the visual encoder and language encoder in a pre-trained visual language model to construct an initial visual language model; performing comparative learning training on the initial visual language model based on the picture-text pair sample set and a preset loss calculation strategy to obtain a target visual language model; if a user voice command is received, converting the user voice command into text data; and obtaining a target object image corresponding to the text data based on the current environment image, text data and target visual language model collected in real time. The present invention can train a target visual language model adapted to the elderly care scene and can improve the target recognition accuracy of the visual language model in the elderly care scene. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0023] Figure 1 A schematic diagram of a flow chart of a visual language alignment enhancement method provided by an embodiment of the present invention;
[0024] Figure 2 yes Figure 1 A schematic flow chart of a specific implementation of step S13;
[0025] Figure 3 A schematic block diagram of a visual language alignment enhancement device provided by an embodiment of the present invention;
[0026] Figure 4 A schematic block diagram of a computer device provided in accordance with an embodiment of the present invention. DETAILED DESCRIPTION
[0027] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.
[0028] It will be understood that when used in this specification and the appended claims, the terms “comprises” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.
[0029] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the present invention. As used in the specification and appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.
[0030] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
[0031] An embodiment of the present invention provides a visual language alignment enhancement method, device, equipment and medium. The executor of the visual language alignment enhancement method provided in this embodiment can be a visual language alignment enhancement device corresponding to the method, and the visual language alignment enhancement device can be integrated into an elderly care assistance robot.
[0032] See also Figure 1 and Figure 2 , Figure 1 A flowchart of a visual language alignment enhancement method provided by an embodiment of the present invention is shown below. Figure 2 yes Figure 1 FIG. 1 is a flow chart of a specific implementation of step S13. In an embodiment of the present invention, the perspective of the enhancement device will be described based on the visual language alignment.
[0033] like Figure 1 As shown, the visual language alignment enhancement method provided by the embodiment of the present invention includes the following steps S11 to S15.
[0034] S11 . Obtain an object image sample set and a text sample set, and construct an image-text pair sample set based on the object image sample set and the text sample set.
[0035] In this embodiment, the object image sample set includes several object image samples, and each object image sample is a typical object image in the elderly care scene, such as an image of a medicine bottle, a walker, a blood pressure monitor, reading glasses, and the like. The text sample set includes a text sample (i.e., a language description) that matches each object image sample. These text samples can be instructions, questions, and fuzzy phrases in the elderly care scene, such as "the medicine I take in the morning," "the medicine in the white bottle," "my high blood pressure medicine," and the like. Typical object images in the elderly care scene and language descriptions that match the typical object images can be collected by an elderly care assistance robot. After obtaining the object image sample set and the text sample set, a picture-text pair sample set is constructed based on the object image sample set and the text sample set, and the picture-text pair sample set is used as training data for the subsequent training model.
[0036] Specifically, for each object image sample in the object image sample set, the object image sample and the text sample in the text sample set that matches the object image sample are combined into a positive image-text pair sample, and the object image sample and multiple text samples in the text sample set that do not match the object image sample (i.e., the remaining text samples in the text sample set except for the text sample that matches the object image sample) are respectively combined into corresponding multiple negative image-text pair samples. The positive image-text pair sample and multiple negative image-text pair samples corresponding to each object image sample form an image-text pair sample set. It can also be understood that the image-text pair sample set includes a positive image-text pair sample and multiple negative image-text pair samples corresponding to each object image sample, wherein the positive image-text pair sample includes the object image sample and the text sample that matches the object image sample, for example, the object image sample is an image of "antihypertensive medication used by the elderly" and the text sample is "the medicine I took this morning"; the negative image-text pair sample includes the object image sample and the text sample that does not match the object image sample, for example, the object image sample is an image of "antihypertensive medication used by the elderly" and the text sample is "the book I just read". The image-text pair sample set contains positive image-text pair samples and negative image-text pair samples, which is helpful in enhancing the model's comparative learning effect and improving the model's ability to discriminate fine-grained semantic alignment during subsequent model training.
[0037] S12. Add the adaptation module and the projection head to the visual encoder and language encoder in the pre-trained visual language model to construct an initial visual language model.
[0038] In this embodiment, a pre-trained visual language model is used as the basic model, wherein the pre-trained visual language model can be CLIP, BLIP, MiniGPT, etc. A trainable adaptation module is inserted into the middle layer of the visual encoder and the language encoder in the pre-trained visual language model, and a group of projection heads are added after the output of the adaptation module, thereby adjusting the structure of the pre-trained visual language model to obtain an initial visual language model. Among them, the adaptation module is composed of a Bottleneck structure, the adaptation module in the visual encoder is responsible for remapping general image features into image features in the elderly care scenario, and the adaptation module in the language encoder is responsible for remapping general text features into text features in the elderly care scenario. The projection head added after the adaptation module can respectively map the image features and text features to a shared embedding space, so as to calculate the similarity between images and texts through comparative learning.
[0039] S13. Perform comparative learning training on the initial visual language model based on the image-text pair sample set and a preset loss calculation strategy to obtain a target visual language model.
[0040] In this embodiment, the image-text pair sample set is training data constructed for the elderly care scenario. The initial visual language model is trained by comparative learning using the image-text pair sample set and a preset loss calculation strategy. The initial visual language model is targeted for domain adaptation and multimodal alignment optimization based on the characteristics of the elderly care scenario to solve the problem of semantic disconnection between the initial visual language model and the elderly care scenario, and to improve the recognition accuracy and responsiveness of the initial visual language model for low-frequency elderly care objects (such as hearing aids, specific medicines, etc.) in the elderly care scenario, thereby training the target visual language model. The target visual language model can more accurately understand the language descriptions of elderly users, improve the naturalness of interaction and fault tolerance. Specifically, considering the problems of edge deployment and limited training resources, a partial fine-tuning strategy can be used to train the initial visual language model, that is, the VLM backbone structure is frozen and remains unchanged, and only the adaptation module and projection head in the initial visual language model are trained. The parameters of the adaptation module and projection head are optimized by combining the image-text pair sample set and the loss calculation strategy to achieve domain alignment optimization under low-resource conditions.
[0041] In one embodiment, if Figure 2 As shown, step S13 includes:
[0042] S131, selecting an object image sample from the image-text pair sample set, and inputting a positive image-text pair sample and a plurality of negative image-text pair samples containing the object image sample from the image-text pair sample set into the initial visual language model;
[0043] S132, performing feature extraction on the object image sample using the visual encoder in the initial visual language model to obtain image features corresponding to the object image sample;
[0044] S133, performing feature extraction on the text sample in the positive image-text pair sample and each of the multiple negative image-text pair samples using a language encoder in the initial visual language model to obtain a positive text feature corresponding to the positive image-text pair sample and a negative text feature corresponding to each negative image-text pair sample;
[0045] S134, calculating the similarity between the image feature and the positive text feature and the negative text feature corresponding to each negative image-text pair sample through the contrastive learning module in the initial visual language model, to obtain the positive sample similarity corresponding to the positive image-text pair sample and the negative sample similarity corresponding to each negative image-text pair sample;
[0046] S135, calculating a loss value based on the negative text features and negative sample similarity corresponding to each negative image-text pair sample, the positive text features, the positive sample similarity, and the loss calculation strategy;
[0047] S136, optimizing the parameters of the adaptation module and the projection head in the initial visual language model using the loss value to obtain an optimized initial visual language model;
[0048] S137. Update the initial visual language model using the optimized initial visual language model, and return to the step of selecting an object image sample from the image-text pair sample set, and inputting a positive image-text pair sample and multiple negative image-text pair samples containing the object image sample in the image-text pair sample set into the initial visual language model until the loss value meets the preset stopping condition, then stop training and output the target visual language model.
[0049] In this embodiment, when training the initial visual language model, first, an object image sample is selected from the image-text pair sample set, wherein an object image sample can be selected randomly or sequentially, and a positive image-text pair sample and multiple negative image-text pair samples containing the object image sample in the image-text pair sample set are input into the initial visual language model, and the initial visual language model is used to calculate the image-text pair similarity of the positive image-text pair sample and the multiple negative image-text pair samples. Specifically, the initial visual language model includes a visual encoder, a language encoder and a contrastive learning module. The visual encoder is used to extract features of object image samples to obtain image features corresponding to the object image samples; the language encoder is used to extract features of text samples in each of the positive image-text pair samples and multiple negative image-text pair samples, thereby obtaining positive text features corresponding to the positive image-text pair samples and negative text features corresponding to each negative image-text pair sample; the contrastive learning module is used to calculate the similarity between the image features and the positive text features to obtain the positive sample similarity corresponding to the positive image-text pair samples, and the contrastive learning module is used to calculate the similarity between the image features and the negative text features corresponding to each negative image-text pair sample to obtain the negative sample similarity corresponding to each negative image-text pair sample.
[0050] Then, the loss value is calculated based on the negative text features and negative sample similarity, positive text features, positive sample similarity, and loss calculation strategy corresponding to each negative image-text pair sample. Among them, the loss calculation strategy can adaptively adjust the importance of each negative image-text pair sample based on the semantic distance between the negative text features and the positive text features corresponding to each negative image-text pair sample, thereby maximizing the relative importance of the positive image-text pair sample. Using the loss value to optimize only the parameters of the adaptation module and projection head in the initial visual language model can effectively reduce the risk of misjudgment caused by semantic ambiguity, improve the model's fault tolerance for ambiguous semantics in elderly care scenarios, and enable the model to complete image-text alignment more accurately. In addition, only the adaptation module and projection head in the initial visual language model are trained, without the need to adjust all parameters in the initial visual language model. This can improve training efficiency, thereby quickly completing this round of fine-tuning training of the initial visual language model and obtaining an optimized initial visual language model.
[0051] Finally, the optimized initial visual language model is used to update the initial visual language model, and the process returns to step S131 to begin the next round of training for the initial visual language model. The training process is repeated until the loss value meets a preset stopping condition. Training is then stopped and the initial visual language model at this point is output as the final target visual language model. The preset stopping condition can be when the loss value drops to a certain level and stabilizes, or when the number of training rounds corresponding to the loss value reaches a preset number, which is not specifically limited here.
[0052] In one embodiment, step S135 includes:
[0053] The semantic adjustment weight value corresponding to each negative image-text pair sample is calculated according to the negative text feature corresponding to each negative image-text pair sample, the positive text feature and the semantic adjustment weight function in the loss calculation strategy;
[0054] The loss value is obtained by calculating the negative sample similarity and semantic adjustment weight value corresponding to each negative image-text pair sample, the positive sample similarity, and the loss function in the loss calculation strategy.
[0055] In this embodiment, in the elderly care scenario, language descriptions are often semantically ambiguous or polysemous. For example, the language description "my medicine" may refer to multiple similar descriptions such as "antihypertensive medicine" and "hypoglycemic medicine". These similar descriptions are not truly unrelated, but are semantically close "hard negatives" (i.e., hard negative samples). If the standard InfoNCE (Noise Contrastive Estimation) loss function is still used to forcibly "push away" the hard negative samples (i.e., increase the difference between the hard negative samples and the anchor samples), it is easy to cause training oscillation or even overfitting. In order to solve this problem, this embodiment first calculates the semantic adjustment weight value corresponding to each negative image-text pair sample through the negative text features, positive text features and adjustment weight function corresponding to each negative image-text pair sample, wherein the semantic adjustment weight value reflects the semantic distance between the positive text feature and the corresponding negative text feature; then, the positive sample similarity and the negative sample similarity and semantic adjustment weight value corresponding to each negative image-text pair sample are substituted into the loss function to calculate the loss value. The loss function can adaptively adjust the importance of each negative sample similarity based on the semantic adjustment weight value corresponding to each negative image-text pair sample, that is, adaptively adjust the importance of each negative image-text pair sample, and then maximize the relative importance of the positive image-text pair sample, so that after the initial visual language model is optimized using the loss value, the model's tolerance to fuzzy semantics can be improved, effectively reducing the risk of misjudgment caused by semantic ambiguity, and allowing the model to complete image-text alignment more accurately.
[0056] In one embodiment, the semantic adjustment weight function is Among them, w j is the semantic adjustment weight value corresponding to the jth negative image-text pair sample, λ is the preset eccentricity parameter, t is the positive text feature, is the negative text feature corresponding to the jth negative image-text pair sample, and N is the total number of the multiple negative image-text pair samples.
[0057] In this embodiment, the semantic adjustment weight function is used to calculate the semantic adjustment weight value corresponding to each negative image-text pair sample based on the positive text feature and the negative text feature corresponding to each negative image-text pair sample. Wherein, λ is a preset eccentricity parameter that can be set based on experience. is the cosine similarity function, which is used to calculate the positive text feature t and the negative text feature The similarity between them. Semantic adjustment weight value w j Reflects the positive text feature t and the negative text feature The semantic distance between them.
[0058] In one embodiment, the loss function is Among them, L improved is the loss value, S is the positive sample similarity, τ is the preset temperature parameter, w j is the semantic adjustment weight value corresponding to the j-th negative image-text pair sample, is the negative sample similarity corresponding to the j-th negative image-text pair sample, and N is the total number of the multiple negative image-text pair samples.
[0059] In this embodiment, the loss function is used to calculate the loss value based on the positive sample similarity and the negative sample similarity and semantic adjustment weight value corresponding to each negative image-text pair sample. By adding the semantic adjustment weight value, the importance of each negative sample similarity is adaptively adjusted based on the semantic adjustment weight value corresponding to each negative image-text pair sample.
[0060] S14: If a user voice instruction is received, convert the user voice instruction into text data.
[0061] In this embodiment, after the target visual language model is trained, during specific use, when a user voice command is received, such as the voice of "help me get the white bottle of medicine", the user voice command is converted into text data through voice recognition, so that the text data can be subsequently input into the target visual language model for processing.
[0062] S15 . Obtain a target object image corresponding to the text data based on the current environment image collected in real time, the text data, and the target visual language model.
[0063] In this embodiment, the current environment image collected in real time is obtained, the current environment image collected in real time and text data are input into the target visual language model, and target recognition is performed through the target visual language model to obtain the target object image corresponding to the text data, so that the task corresponding to the user's voice command can be completed subsequently based on the recognized target object image.
[0064] The executor of the visual language alignment enhancement method of this embodiment is the visual language alignment enhancement device corresponding to the method. The visual language alignment enhancement device is integrated into an elderly care assistance robot. The elderly care assistance robot can capture the current environment image in real time through a camera or other sensors. For example, the text data is the text "Help me get the white bottle of medicine". The text data and the current environment image collected in real time by the elderly care assistance robot are input into the target visual language model for target recognition, and an image of "antihypertensive medicine" matching the text "Help me get the white bottle of medicine" is obtained. The elderly care assistance robot can obtain the corresponding antihypertensive medicine for the user based on the "antihypertensive medicine" image.
[0065] In one embodiment, step S15 includes:
[0066] Inputting the current environment image and the text data into the target visual language model to obtain the similarity of the current image-text pair;
[0067] If it is determined that the similarity between the current image and text pair is greater than or equal to a preset threshold, the current environment image is used as the target object image.
[0068] In this embodiment, the current environment image and text data are input into the target visual language model to calculate feature similarity, thereby obtaining the current image-text pair similarity. If the current image-text pair similarity is determined to be greater than or equal to a preset threshold, indicating a high model recognition confidence, the current environment image is then used as the target object image.
[0069] In one embodiment, after the step of inputting the current environment image and the text data into the target visual language model to obtain the current image-text pair similarity, the method further includes:
[0070] If it is determined that the similarity between the current image and text pair is less than the preset threshold, an interactive confirmation feedback mode is triggered, and the target object image is obtained based on the interactive confirmation feedback mode.
[0071] In this embodiment, if it is determined that the similarity of the current image-text pair is less than a preset threshold, indicating that the model recognition confidence is low, the interactive confirmation feedback mode will be triggered, and the target object image will be obtained based on the interactive confirmation feedback mode, thereby improving the target recognition accuracy.
[0072] In one embodiment, obtaining the target object image based on the interactive confirmation feedback mode includes: generating user confirmation prompt information based on the interactive confirmation feedback mode; receiving user confirmation feedback information corresponding to the user confirmation prompt information; if the user confirmation feedback information is a positive feedback result, using the current environment image as the target object image; if the user confirmation feedback information is a negative feedback result, obtaining a re-captured new current environment image, updating the current environment image using the new current environment image, and returning to execute the step of inputting the current environment image and the text data into the target visual language model to obtain the similarity of the current image-text pair.
[0073] In this embodiment, after the interactive confirmation feedback mode is triggered, the system will generate a user confirmation prompt message, such as "Excuse me, are you talking about white antihypertensive medicine?" This user confirmation prompt message can be output to the user through voice or displayed on the screen, and then the user confirmation feedback message is received, thereby forming a closed-loop interaction. Furthermore, if the user confirmation feedback message is a positive feedback result, such as "yes", it means that although the feature similarity between the current environment image and the text data is low, the positive feedback result can determine that the user has confirmed that the current environment image matches the text data, and the current environment image is used as the target object image. If the user confirmation feedback message is a negative feedback result, such as "no", the current environment image is collected and updated, and the target recognition is re-performed through the target visual language model.
[0074] In one embodiment, if it is determined that the similarity of the current image-text pair is less than the preset threshold, the interactive confirmation feedback mode is triggered, and after the step of obtaining the target object image based on the interactive confirmation feedback mode, it also includes: forming a positive image-text pair with the text data and the target object image and storing it; if it is determined that the number of stored positive image-text pairs reaches the preset number, constructing a new image-text pair sample set based on the stored positive image-text pairs, updating the image-text pair sample set using the new image-text pair sample set, and returning to step S13. In this embodiment, the target object image obtained in the interactive confirmation feedback mode can be formed into a positive image-text pair with the text data and stored, so that the target visual language model can be updated subsequently through these stored positive image-text pairs, and lightweight continuous learning can be performed through user confirmation feedback, and the representation distribution of image-text pairs in the elderly care scenario can be continuously updated to improve the long-term adaptability of the target visual language model, support continuous expansion and updating, so as to adapt to the unique living habits of different regions and individual users and improve the personalization ability of the model.
[0075] The visual language alignment enhancement method disclosed in the present invention performs targeted domain adaptation and multimodal alignment optimization on the initial visual language model based on the characteristics of the elderly care scenario, and then trains a target visual language model adapted to the elderly care scenario, which can improve the target recognition accuracy of the visual language model in the elderly care scenario.
[0076] The embodiment of the present invention further provides a visual language alignment enhancement device, which is used to perform any embodiment of the aforementioned visual language alignment enhancement method. The visual language alignment enhancement device can be integrated into an elderly care assistance robot. Figure 3 , Figure 3 This is a schematic block diagram of a visual language alignment enhancement device according to an embodiment of the present invention. The visual language alignment enhancement device according to an embodiment of the present invention includes a sample construction unit 11, a model construction unit 12, a model training unit 13, a speech processing unit 14, and an object recognition unit 15. The functional units are described in detail as follows:
[0077] The sample construction unit 11 is used to obtain an object image sample set and a text sample set, and construct an image-text pair sample set based on the object image sample set and the text sample set;
[0078] A model construction unit 12 is configured to add the adaptation module and the projection head to the visual encoder and the language encoder in the pre-trained visual language model to construct an initial visual language model;
[0079] A model training unit 13 is configured to perform comparative learning training on the initial visual language model based on the image-text pair sample set and a preset loss calculation strategy to obtain a target visual language model;
[0080] The voice processing unit 14 is configured to convert a user voice command into text data upon receiving the user voice command;
[0081] The target recognition unit 15 is configured to obtain a target object image corresponding to the text data based on a current environment image collected in real time, the text data, and the target visual language model.
[0082] In one embodiment, the model training unit 13 is specifically configured to:
[0083] selecting an object image sample from the image-text pair sample set, and inputting a positive image-text pair sample and a plurality of negative image-text pair samples containing the object image sample from the image-text pair sample set into the initial visual language model;
[0084] Extracting features from the object image sample using a visual encoder in the initial visual language model to obtain image features corresponding to the object image sample;
[0085] Performing feature extraction on the text sample in the positive image-text pair sample and each of the multiple negative image-text pair samples using a language encoder in the initial visual language model to obtain a positive text feature corresponding to the positive image-text pair sample and a negative text feature corresponding to each negative image-text pair sample;
[0086] The contrastive learning module in the initial visual language model calculates the similarity between the image feature and the positive text feature and the negative text feature corresponding to each negative image-text pair sample, thereby obtaining the positive sample similarity corresponding to the positive image-text pair sample and the negative sample similarity corresponding to each negative image-text pair sample;
[0087] Calculating a loss value based on the negative text features and negative sample similarity corresponding to each negative image-text pair sample, the positive text features, the positive sample similarity, and the loss calculation strategy;
[0088] Optimizing the parameters of the adaptation module and the projection head in the initial visual language model using the loss value to obtain an optimized initial visual language model;
[0089] The optimized initial visual language model is used to update the initial visual language model, and the step of selecting an object image sample from the image-text pair sample set and inputting a positive image-text pair sample and multiple negative image-text pair samples containing the object image sample in the image-text pair sample set into the initial visual language model is returned to the step of inputting a positive image-text pair sample and multiple negative image-text pair samples containing the object image sample in the image-text pair sample set into the initial visual language model until the loss value meets a preset stopping condition, then stopping the training and outputting the target visual language model.
[0090] In one embodiment, when executing the step of calculating the loss value based on the negative text features and negative sample similarity corresponding to each negative image-text pair sample, the positive text features, the positive sample similarity, and the loss calculation strategy, the model training unit 13 is specifically configured to:
[0091] The semantic adjustment weight value corresponding to each negative image-text pair sample is calculated according to the negative text feature corresponding to each negative image-text pair sample, the positive text feature and the semantic adjustment weight function in the loss calculation strategy;
[0092] The loss value is obtained by calculating the negative sample similarity and semantic adjustment weight value corresponding to each negative image-text pair sample, the positive sample similarity, and the loss function in the loss calculation strategy.
[0093] In one embodiment, the semantic adjustment weight function is Among them, w j is the semantic adjustment weight value corresponding to the jth negative image-text pair sample, λ is the preset eccentricity parameter, t is the positive text feature, is the negative text feature corresponding to the jth negative image-text pair sample, and N is the total number of the multiple negative image-text pair samples.
[0094] In one embodiment, the loss function is Among them, L improved is the loss value, S is the positive sample similarity, τ is the preset temperature parameter, w j is the semantic adjustment weight value corresponding to the j-th negative image-text pair sample, is the negative sample similarity corresponding to the j-th negative image-text pair sample, and N is the total number of the multiple negative image-text pair samples.
[0095] In one embodiment, the target recognition unit 15 is specifically configured to:
[0096] Inputting the current environment image and the text data into the target visual language model to obtain the similarity of the current image-text pair;
[0097] If it is determined that the similarity between the current image and text pair is greater than or equal to a preset threshold, the current environment image is used as the target object image.
[0098] In one embodiment, after executing the step of inputting the current environment image and the text data into the target visual language model to obtain the current image-text pair similarity, the target recognition unit 15 is further configured to:
[0099] If it is determined that the similarity between the current image and text pair is less than the preset threshold, an interactive confirmation feedback mode is triggered, and the target object image is obtained based on the interactive confirmation feedback mode.
[0100] In one embodiment, when the target recognition unit 15 executes the step of obtaining the target object image based on the interactive confirmation feedback mode, it is specifically used to: generate user confirmation prompt information based on the interactive confirmation feedback mode; receive user confirmation feedback information corresponding to the user confirmation prompt information; if the user confirmation feedback information is a positive feedback result, use the current environment image as the target object image; if the user confirmation feedback information is a negative feedback result, obtain a newly captured current environment image, update the current environment image using the new current environment image, and return to execute the step of inputting the current environment image and the text data into the target visual language model to obtain the similarity of the current image-text pair.
[0101] In one embodiment, after executing the step of triggering the interactive confirmation feedback mode if it is determined that the similarity of the current image-text pair is less than the preset threshold, and obtaining the target object image based on the interactive confirmation feedback mode, the target recognition unit 15 is also used to: form a positive image-text pair with the text data and the target object image and store them; if it is determined that the number of stored positive image-text pairs reaches a preset number, construct a new image-text pair sample set based on the stored positive image-text pairs, update the image-text pair sample set using the new image-text pair sample set, and return to the model training unit 13 to execute the step of performing comparative learning and training on the initial visual language model based on the image-text pair sample set and the preset loss calculation strategy to obtain a target visual language model.
[0102] The present invention discloses a visual language alignment enhancement device, which is used to execute any embodiment of the aforementioned visual language alignment enhancement method. By performing targeted domain adaptation and multimodal alignment optimization on the initial visual language model based on the characteristics of the elderly care scenario, and then training a target visual language model adapted to the elderly care scenario, the target recognition accuracy of the visual language model in the elderly care scenario can be improved. The visual language alignment enhancement device can be deployed in elderly care assistance robots on different platforms, has strong compatibility, and is easy to integrate.
[0103] The above-mentioned visual language alignment enhancement method can be implemented in the form of a computer program. The computer program can be used in Figure 4 Runs on the computer device shown.
[0104] See also Figure 4 , Figure 4 5 is a schematic block diagram of a computer device according to an embodiment of the present invention. The computer device 500 includes a processor 502 , a memory, and a network interface 505 connected via a device bus 501 , wherein the memory may include a storage medium 503 and an internal memory 504 .
[0105] The storage medium 503 may store an operating device 5031 and a computer program 5032. When the computer program 5032 is executed, the processor 502 may execute a visual language alignment enhancement method.
[0106] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500.
[0107] The internal memory 504 provides an environment for running the computer program 5032 in the storage medium 503 . When the computer program 5032 is executed by the processor 502 , the processor 502 can execute the visual language alignment enhancement method.
[0108] The network interface 505 is used for network communication, such as providing data information transmission. Those skilled in the art will understand that Figure 4 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention and does not constitute a limitation on the computer device 500 to which the solution of the present invention is applied. The specific computer device 500 may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.
[0109] The processor 502 is configured to run a computer program 5032 stored in the memory to implement the visual language alignment enhancement method disclosed in the embodiment of the present invention.
[0110] Those skilled in the art will understand that Figure 4 The embodiment of the computer device shown in the figure does not constitute a limitation on the specific composition of the computer device. In other embodiments, the computer device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, in some embodiments, the computer device may only include a memory and a processor. In such an embodiment, the structure and function of the memory and processor are the same as those in the figure. Figure 4 The embodiments shown are consistent and will not be described again here.
[0111] It should be understood that in the embodiment of the present invention, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.
[0112] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the visual language alignment enhancement method disclosed in an embodiment of the present invention is implemented.
[0113] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the devices and units described above can refer to the corresponding processes in the aforementioned method embodiments, which will not be repeated here. Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0114] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, systems and methods can be implemented in other ways. For example, the system embodiments described above are merely schematic. For example, the division of units is merely a logical function division. In actual implementation, there may be other division methods, and units with the same function may be combined into one unit. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, systems or units, or may be an electrical, mechanical or other form of connection.
[0115] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected based on actual needs to achieve the objectives of the embodiments of the present invention.
[0116] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.
[0117] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a background server, or a network device, etc.) to execute all or part of the steps of the various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), disk or optical disk, etc. Various media that can store program codes.
[0118] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.
Claims
1. A visual language alignment enhancement method, characterized in that: include: Acquire an object image sample set and a text sample set, and construct an image-text pair sample set based on the object image sample set and the text sample set; Add the adaptation module and projection head to the visual encoder and language encoder in the pre-trained visual language model to construct an initial visual language model; Performing comparative learning training on the initial visual language model based on the image-text pair sample set and a preset loss calculation strategy to obtain a target visual language model; If a user voice command is received, converting the user voice command into text data; A target object image corresponding to the text data is obtained based on a current environment image collected in real time, the text data and the target visual language model.
2. The visual language alignment enhancement method according to claim 1, characterized in that: The step of performing comparative learning and training on the initial visual language model based on the image-text pair sample set and a preset loss calculation strategy to obtain a target visual language model includes: selecting an object image sample from the image-text pair sample set, and inputting a positive image-text pair sample and a plurality of negative image-text pair samples containing the object image sample from the image-text pair sample set into the initial visual language model; Extracting features from the object image sample using a visual encoder in the initial visual language model to obtain image features corresponding to the object image sample; Performing feature extraction on the text sample in the positive image-text pair sample and each of the multiple negative image-text pair samples using a language encoder in the initial visual language model to obtain a positive text feature corresponding to the positive image-text pair sample and a negative text feature corresponding to each negative image-text pair sample; The contrastive learning module in the initial visual language model calculates the similarity between the image feature and the positive text feature and the negative text feature corresponding to each negative image-text pair sample, thereby obtaining the positive sample similarity corresponding to the positive image-text pair sample and the negative sample similarity corresponding to each negative image-text pair sample; Calculating a loss value based on the negative text features and negative sample similarity corresponding to each negative image-text pair sample, the positive text features, the positive sample similarity, and the loss calculation strategy; Optimizing the parameters of the adaptation module and the projection head in the initial visual language model using the loss value to obtain an optimized initial visual language model; The optimized initial visual language model is used to update the initial visual language model, and the step of selecting an object image sample from the image-text pair sample set and inputting a positive image-text pair sample and multiple negative image-text pair samples containing the object image sample in the image-text pair sample set into the initial visual language model is returned to the step of inputting a positive image-text pair sample and multiple negative image-text pair samples containing the object image sample in the image-text pair sample set into the initial visual language model until the loss value meets a preset stopping condition, then stopping the training and outputting the target visual language model.
3. The visual language alignment enhancement method according to claim 2, characterized in that: The loss value is calculated according to the negative text features and negative sample similarity corresponding to each negative image-text pair sample, the positive text features, the positive sample similarity, and the loss calculation strategy, including: The semantic adjustment weight value corresponding to each negative image-text pair sample is calculated according to the negative text feature corresponding to each negative image-text pair sample, the positive text feature and the semantic adjustment weight function in the loss calculation strategy; The loss value is obtained by calculating the negative sample similarity and semantic adjustment weight value corresponding to each negative image-text pair sample, the positive sample similarity, and the loss function in the loss calculation strategy.
4. The visual language alignment enhancement method according to claim 3, characterized in that: The semantic adjustment weight function is Among them, w j is the semantic adjustment weight value corresponding to the jth negative image-text pair sample, λ is the preset eccentricity parameter, t is the positive text feature, is the negative text feature corresponding to the jth negative image-text pair sample, and N is the total number of the multiple negative image-text pair samples.
5. The visual language alignment enhancement method according to claim 3, characterized in that: The loss function is Among them, L improved is the loss value, S is the positive sample similarity, τ is the preset temperature parameter, w j is the semantic adjustment weight value corresponding to the j-th negative image-text pair sample, is the negative sample similarity corresponding to the j-th negative image-text pair sample, and N is the total number of the multiple negative image-text pair samples.
6. The visual language alignment enhancement method according to claim 1, characterized in that: The step of obtaining a target object image corresponding to the text data based on the current environment image collected in real time, the text data, and the target visual language model includes: Inputting the current environment image and the text data into the target visual language model to obtain the similarity of the current image-text pair; If it is determined that the similarity between the current image and text pair is greater than or equal to a preset threshold, the current environment image is used as the target object image.
7. The visual language alignment enhancement method according to claim 6, characterized in that: After the step of inputting the current environment image and the text data into the target visual language model to obtain the similarity of the current image-text pair, the method further includes: If it is determined that the similarity between the current image and text pair is less than the preset threshold, an interactive confirmation feedback mode is triggered, and the target object image is obtained based on the interactive confirmation feedback mode.
8. A visual language alignment enhancement device, characterized in that: include: A sample construction unit, configured to obtain an object image sample set and a text sample set, and construct an image-text pair sample set based on the object image sample set and the text sample set; A model construction unit is used to add the adaptation module and projection head to the visual encoder and language encoder in the pre-trained visual language model to construct an initial visual language model; a model training unit, configured to perform comparative learning training on the initial visual language model based on the image-text pair sample set and a preset loss calculation strategy to obtain a target visual language model; A voice processing unit, configured to convert a user voice instruction into text data upon receiving the user voice instruction; The target recognition unit is used to obtain a target object image corresponding to the text data based on the current environment image collected in real time, the text data and the target visual language model.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the visual language alignment enhancement method according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the visual language alignment enhancement method according to any one of claims 1 to 7.