Visual language positioning method and device based on multiple modes

Abstract

The invention discloses a visual language positioning method and device based on multiple modes. The multi-mode-based visual language positioning method comprises the following steps: obtaining people searching instruction information obtained through interaction with a user; acquiring to-be-recognized head portrait information; acquiring a user-defined gesture information base; generating indication information; acquiring video stream information and voice information fed back by the user; obtaining first text information; obtaining second text information; inputting the first text information and the second text information into a trained semantic understanding model to obtain target object text information; and video stream information in the nursing home is retrieved, so that the target object is positioned in the video stream information, and image information of the target object is acquired. According to the method, the types of provided semantic information are enriched, and voice information is not purely used for positioning auxiliary features, so that the positioning auxiliary features can be provided by using the method provided by the invention even if the elderly are unclear in speech or heavier in accent.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a multi-modal visual language localization method and a multi-modal visual language localization device. Background Art

[0002] Visual language understanding and localization, referred to as visual grounding (VG), also known as reference expression comprehension (REC) or phrase grounding (PG), refers to locating the area of ​​the bounding box described by the text expression in a specific image. This technology has become one of the key technologies for artificial intelligence human-computer interaction. Due to the cross-modal nature of the visual language understanding and localization model, it is necessary to understand the semantics of the language expression and the image at the same time, which has always been a complex and challenging task.

[0003] In nursing homes, the elderly usually have a strong accent and may not speak clearly. Therefore, if you want to use visual language to understand and locate in such an environment, it is difficult to accurately locate the target by relying solely on the elderly’s language expression.

[0004] Therefore, it is desired to have a technical solution to overcome or at least alleviate at least one of the above-mentioned defects of the prior art.

[0005] Application Contents

[0006] The purpose of the present application is to provide a multimodal visual language localization method to overcome or at least alleviate at least one of the above-mentioned defects of the prior art.

[0007] To achieve the above objectives, the present application provides a multimodal visual language localization method for interacting with the elderly in a nursing home. The multimodal visual language localization method includes:

[0008] Acquire the person-searching instruction information obtained through interaction with the user;

[0009] Acquire the head portrait information to be identified according to the person-searching instruction information;

[0010] Obtain a custom gesture information library based on the avatar information to be recognized;

[0011] Generate instruction information;

[0012] Obtaining video stream information and voice information fed back by the user according to the instruction information;

[0013] Acquire first text information according to the custom gesture information library and the video stream information;

[0014] Acquire second text information according to the voice information;

[0015] Get the trained semantic understanding model;

[0016] Inputting the first text information and the second text information into a trained semantic understanding model to obtain target object text information;

[0017] The video stream information transmitted by one or more camera devices in the nursing home is retrieved according to the target object text information, so as to locate the target object in the video stream information and obtain the image information of the target object.

[0018] Optionally, acquiring the head portrait information to be identified according to the person-searching instruction information includes:

[0019] When the person-searching instruction information is obtained, image request information is generated and sent to the user;

[0020] Obtaining the answer information fed back by the user according to the image request information;

[0021] When the answer information is yes, a camera device of the human-machine interaction device generates a shooting signal;

[0022] Acquire the head portrait information to be identified that is photographed by the camera device of the human-computer interaction device according to the shooting signal.

[0023] Optionally, acquiring a custom gesture information library according to the avatar information to be identified includes:

[0024] Obtaining an avatar gesture library corresponding table, wherein the avatar gesture library corresponding table includes at least one preset avatar and a custom gesture information library corresponding to each preset avatar, wherein the custom gesture information library includes at least one preset custom gesture, and each preset custom gesture corresponds to a sentence information or word information;

[0025] The similarity between the head portrait information to be identified and each preset head portrait is calculated respectively, so as to obtain a custom gesture information library corresponding to the preset head portraits whose similarity exceeds a preset threshold.

[0026] Optionally, the acquiring the first text information according to the custom gesture information library and the video stream information includes:

[0027] Parsing the video stream information to obtain gesture information in the video stream information;

[0028] Sentence information or word information corresponding to the preset custom gestures identical to each gesture information is respectively acquired from the custom gesture information library, and each of the sentence information or word information constitutes the first text information.

[0029] Optionally, the trained semantic understanding model is a BERT model.

[0030] Optionally, the retrieving video stream information transmitted by one or more camera devices in the nursing home according to the target object text information, thereby locating the target object in the video stream information and acquiring image information of the target object includes:

[0031] Identify the target object text information, determine whether the camera position and time period information can be determined from the target object text information, and if so,

[0032] Acquire a camera position database, wherein the camera position database includes at least one preset camera position and a camera corresponding to each preset camera position;

[0033] Acquire a camera device corresponding to a preset camera device position that is the same as the camera device position, the camera device being the designated camera device;

[0034] Obtaining video stream information of a specified camera device within a time period;

[0035] The target object is located for each frame of image information in the video stream information according to the target object text information, and when the target object is located, the image information of the target object is obtained.

[0036] Optionally, the step of retrieving video stream information transmitted by one or more camera devices in the nursing home according to the target object text information, thereby locating the target object in the video stream information and acquiring image information of the target object further includes:

[0037] Identify the target object text information, determine whether the camera position and time period information can be determined from the target object text information, and if not,

[0038] Determine whether the camera position can be determined from the target object text information, and if so,

[0039] Acquire a camera position database, wherein the camera position database includes at least one preset camera position and a camera corresponding to each preset camera position;

[0040] Acquire a camera device corresponding to a preset camera device position that is the same as the camera device position, the camera device being the designated camera device;

[0041] Generate time period query information, wherein the time period query information includes multiple time period options;

[0042] Obtaining a time period option selected by the user according to the time period inquiry information;

[0043] Get the video stream information of the specified camera device within the time period option;

[0044] The target object is located for each frame of image information in the video stream information according to the target object text information, and when the target object is located, the image information of the target object is obtained.

[0045] Optionally, the step of retrieving video stream information transmitted by one or more camera devices in the nursing home according to the target object text information, thereby locating the target object in the video stream information and acquiring image information of the target object further includes:

[0046] Identify the target object text information, determine whether the camera position and time period information can be determined from the target object text information, and if not,

[0047] Determine whether the camera position can be determined from the target object text information, if not, then

[0048] Determine whether the time period information can be determined from the target object text information. If so,

[0049] Acquire video stream information of a camera device closest to the user within the time period;

[0050] The target object is located for each frame of image information in the video stream information according to the target object text information, and when the target object is located, the image information of the target object is obtained.

[0051] Optionally, when the target object cannot be located in the acquired video stream information, the multimodal visual language localization method further includes:

[0052] The adjacent camera devices to the camera device are selected in sequence through the adjacent selection method for identification until the target object is located.

[0053] The present application also provides a multi-modal visual language positioning device, the multi-modal visual language positioning device comprising:

[0054] A person-searching instruction information acquisition module, the person-searching instruction information acquisition module is used to acquire person-searching instruction information acquired through interaction with a user;

[0055] A module for acquiring head portrait information to be identified, the module for acquiring head portrait information to be identified is used to acquire head portrait information to be identified according to the person-searching instruction information;

[0056] A custom gesture information library acquisition module, wherein the custom gesture information library acquisition module is used to acquire a custom gesture information library according to the avatar information to be identified;

[0057] An indication information generating module, wherein the indication information generating module is used to generate indication information;

[0058] A video stream information and voice information acquisition module, which is used to acquire the video stream information and voice information fed back by the user according to the instruction information;

[0059] A first text information acquisition module, the first text information acquisition module is used to acquire first text information according to the custom gesture information library and the video stream information;

[0060] A second text information acquisition module, the second text information acquisition module is used to acquire second text information according to the voice information;

[0061] A semantic understanding model acquisition module, wherein the semantic understanding model acquisition module is used to acquire a trained semantic understanding model;

[0062] A target object text information acquisition module, wherein the target object text information acquisition module is used to input the first text information and the second text information into a trained semantic understanding model, thereby acquiring the target object text information;

[0063] A positioning module is used to retrieve the video stream information transmitted by one or more cameras in the nursing home according to the text information of the target object, so as to locate the target object in the video stream information and obtain the image information of the target object.

[0064] The multimodal visual language localization method of the present application obtains the first text information through gestures, obtains the second text information through voice, and obtains the target object text information by identifying the first text information and the second text information, thereby having the following advantages:

[0065] 1. It enriches the types of semantic information provided, and does not simply use voice information for positioning auxiliary features, so that even if the elderly speak unclearly or have a heavy accent, the method of this application can be used to provide positioning auxiliary features.

[0066] 2. Through customization, the elderly can express complex text meanings even with simple gestures. For example, a simple gesture can directly refer to a person's basic features, as long as the basic features are pre-stored in the custom gesture information library. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1It is a flowchart of a multimodal visual language localization method according to an embodiment of the present application.

[0068] Figure 2 It is a schematic diagram of the structure of an electronic device in one embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical scheme and advantages of the implementation of this application clearer, the technical scheme in the embodiment of this application will be described in more detail below in conjunction with the drawings in the embodiment of this application. In the drawings, the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The described embodiments are part of the embodiments of this application, not all of them. The embodiments described below with reference to the drawings are exemplary and are intended to be used to explain this application, and should not be construed as limitations on this application. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. The embodiments of this application are described in detail below in conjunction with the drawings.

[0070] In the description of the present application, it should be understood that the terms "center", "longitudinal", "lateral", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the scope of protection of the present application.

[0071] Figure 1 It is a flowchart of a multimodal visual language localization method according to an embodiment of the present application.

[0072] like Figure 1 The image recognition-based smart community information sharing method shown is based on a multimodal visual language localization method and is used for interacting with the elderly in nursing homes.

[0073] The multimodal visual language localization method includes:

[0074] Acquire the person-searching instruction information obtained through interaction with the user;

[0075] Acquire the head portrait information to be identified according to the person-searching instruction information;

[0076] Obtain a custom gesture information library based on the avatar information to be recognized;

[0077] Generate instruction information;

[0078] Obtaining video stream information and voice information fed back by the user according to the instruction information;

[0079] Acquire first text information according to the custom gesture information library and the video stream information;

[0080] Acquire second text information according to the voice information;

[0081] Get the trained semantic understanding model;

[0082] Inputting the first text information and the second text information into a trained semantic understanding model to obtain target object text information;

[0083] The video stream information transmitted by one or more camera devices in the nursing home is retrieved according to the target object text information, so as to locate the target object in the video stream information and obtain the image information of the target object.

[0084] The multimodal visual language localization method of the present application obtains the first text information through gestures, obtains the second text information through voice, and obtains the target object text information by identifying the first text information and the second text information, thereby having the following advantages:

[0085] 1. It enriches the types of semantic information provided, and does not simply use voice information for positioning auxiliary features, so that even if the elderly speak unclearly or have a heavy accent, the method of this application can be used to provide positioning auxiliary features.

[0086] 2. Through customization, the elderly can express complex text meanings even with simple gestures. For example, a simple gesture can directly refer to a person's basic features, as long as the basic features are pre-stored in the custom gesture information library.

[0087] In this embodiment, the person-finding instruction may be sent by the user during the process of interacting with the interactive system. For example, the user may input "I want to search" through text input or select the person-finding search by clicking an option box.

[0088] In this embodiment, obtaining the head portrait information to be identified according to the person-searching instruction information includes:

[0089] After obtaining the person-searching instruction information, an image request information is generated and sent to the user. In this embodiment, the generated image request information is used to request the user to allow the camera device of the human-computer interaction device to take a photo;

[0090] Obtaining answer information fed back by the user according to the image request information. In this embodiment, the answer information may be a selection made by the user according to a pop-up window provided by the image request information, for example, there is a pop-up window for agreement and a pop-up window for disagreement, and the user clicks on the agreement button, or the user provides answer information through voice or other means;

[0091] When the answer information is yes, a camera device of the human-machine interaction device generates a shooting signal;

[0092] Acquire the head portrait information to be identified photographed by the camera device of the human-computer interaction device according to the shooting signal.

[0093] In this embodiment, obtaining a custom gesture information library according to the avatar information to be recognized includes:

[0094] Obtain a corresponding table of an avatar gesture library, wherein the corresponding table of the avatar gesture library includes at least one preset avatar and a custom gesture information library corresponding to each preset avatar, wherein the custom gesture information library includes at least one preset custom gesture, and each preset custom gesture corresponds to a sentence information or word information; in this embodiment, the preset custom gesture can use international universal sign language, so that if an elderly person with sign language experience can directly use sign language, it can also be a gesture set by the elderly person themselves, for example, a movement from left to right can represent a sentence information or word information, and the sentence information and word information can be preset, for example, a person wearing a hat can correspond to a gesture, or the color green can also correspond to a gesture, so that if there are two gestures, they can be combined into a group, that is, (a person wearing a hat, green).

[0095] The similarity between the head portrait information to be identified and each preset head portrait is calculated respectively, so as to obtain a custom gesture information library corresponding to the preset head portraits whose similarity exceeds a preset threshold.

[0096] In this embodiment, acquiring the first text information according to the custom gesture information library and the video stream information includes:

[0097] Parsing the video stream information to obtain gesture information in the video stream information;

[0098] Sentence information or word information corresponding to the preset custom gestures identical to each gesture information is respectively acquired from the custom gesture information library, and each of the sentence information or word information constitutes the first text information.

[0099] In this embodiment, since the order in which the user performs gestures is uncertain, the first text information obtained may not be fluent. For example, the above (person wearing a hat, green) belongs to a first text information, but it is not very fluent in terms of semantics.

[0100] Therefore, after the first text information and the second text information are input into the trained semantic understanding model, the target object text information is obtained.

[0101] In this embodiment, the target object text information may include the following content:

[0102] A description of the characteristics of the person being sought, e.g., wearing a hat, the color of the hat, the color of the clothing, etc.;

[0103] Find the location of a person (this content may or may not be present);

[0104] The time period information for searching the person (this content may or may not be present).

[0105] In this embodiment, the trained semantic understanding model is a BERT model.

[0106] In this embodiment, the step of retrieving video stream information transmitted by one or more camera devices in the nursing home according to the target object text information, thereby locating the target object in the video stream information and obtaining image information of the target object includes:

[0107] Identify the target object text information, determine whether the camera position and time period information can be determined from the target object text information, and if so,

[0108] Acquire a camera position database, wherein the camera position database includes at least one preset camera position and a camera corresponding to each preset camera position;

[0109] Acquire a camera device corresponding to a preset camera device position that is the same as the camera device position, the camera device being the designated camera device;

[0110] Obtaining video stream information of a specified camera device within a time period;

[0111] The target object is located for each frame of image information in the video stream information according to the target object text information, and when the target object is located, the image information of the target object is obtained.

[0112] In this embodiment, the step of retrieving the video stream information transmitted by one or more camera devices in the nursing home according to the target object text information, thereby locating the target object in the video stream information and obtaining the image information of the target object further includes:

[0113] Identify the target object text information, determine whether the camera position and time period information can be determined from the target object text information, and if not,

[0114] Determine whether the camera position can be determined from the target object text information, and if so,

[0115] Acquire a camera position database, wherein the camera position database includes at least one preset camera position and a camera corresponding to each preset camera position;

[0116] Acquire a camera device corresponding to a preset camera device position that is the same as the camera device position, the camera device being the designated camera device;

[0117] Generate time period query information, wherein the time period query information includes multiple time period options;

[0118] Obtaining a time period option selected by the user according to the time period inquiry information;

[0119] Get the video stream information of the specified camera device within the time period option;

[0120] The target object is located for each frame of image information in the video stream information according to the target object text information, and when the target object is located, the image information of the target object is obtained.

[0121] It is understandable that if there is no time information, the video stream information data is too large. Therefore, the user is required to select at least one interval. In this embodiment, there can be multiple time period options, such as within 1 hour, within 2 hours, etc.

[0122] In this embodiment, the step of retrieving the video stream information transmitted by one or more camera devices in the nursing home according to the target object text information, thereby locating the target object in the video stream information and obtaining the image information of the target object further includes:

[0123] Identify the target object text information, determine whether the camera position and time period information can be determined from the target object text information, and if not,

[0124] Determine whether the camera position can be determined from the target object text information, if not, then

[0125] Determine whether the time period information can be determined from the target object text information. If so,

[0126] Acquire video stream information of a camera device closest to the user within the time period;

[0127] The target object is located for each frame of image information in the video stream information according to the target object text information, and when the target object is located, the image information of the target object is obtained.

[0128] In this embodiment, since the user needs to interact with the human-machine interaction device, and the human-machine interaction device generally does not move, and each human-machine interaction device has its own number, the user's position can be known as long as the number of the preset human-machine interaction device and the position of each human-machine interaction device are known.

[0129] In this embodiment, when the target object cannot be located in the acquired video stream information, the multimodal visual language localization method further includes:

[0130] The adjacent camera devices to the camera device are selected in sequence through the adjacent selection method for identification until the target object is located.

[0131] In this embodiment, the positions of each camera device in the nursing home are fixed, so the distances between the positions of each camera device are also known. For example, assuming that in one embodiment, the two adjacent camera devices to camera device 1 are camera device 2 (adjacent means the closest distance to camera device 1), if the video stream information of camera device 1 cannot be positioned, the video stream information of camera device 2 is obtained. If the video stream information of camera device 2 cannot be positioned either, the video stream information of the camera devices other than camera device 1 that are closest to camera device 2 is obtained.

[0132] In this embodiment, the following method is used to locate the target object in the image information:

[0133] Understand image information and generate corresponding visual features;

[0134] Effectively represent the text information of the target object and generate text features;

[0135] In the process of gradually generating visual features, text features are embedded in sequence and jointly encoded with visual features;

[0136] Use deep visual features and text features to build multimodal graphs, perform cross-modal graph reasoning, and roughly locate target objects;

[0137] The cross-modal attention enhancement module is used to enhance the cross-modal features at different scales and refine the positioning results.

[0138] The present application also provides a multimodal visual language positioning device, which includes a person-searching instruction information acquisition module, a head portrait information acquisition module to be identified, a custom gesture information library acquisition module, an instruction information generation module, a video stream information and voice information acquisition module, a first text information acquisition module, a second text information acquisition module, a semantic understanding model acquisition module, a target object text information acquisition module and a positioning module, wherein:

[0139] The person-searching instruction information acquisition module is used to acquire the person-searching instruction information acquired through interaction with the user;

[0140] The module for acquiring the head portrait information to be identified is used to acquire the head portrait information to be identified according to the person-searching instruction information;

[0141] The custom gesture information library acquisition module is used to acquire the custom gesture information library according to the avatar information to be recognized;

[0142] The indication information generating module is used to generate indication information;

[0143] The video stream information and voice information acquisition module is used to acquire the video stream information and voice information fed back by the user according to the instruction information;

[0144] The first text information acquisition module is used to acquire first text information according to the custom gesture information library and the video stream information;

[0145] The second text information acquisition module is used to acquire the second text information according to the voice information;

[0146] The semantic understanding model acquisition module is used to obtain the trained semantic understanding model;

[0147] The target object text information acquisition module is used to input the first text information and the second text information into a trained semantic understanding model, thereby acquiring the target object text information;

[0148] The positioning module is used to retrieve the video stream information transmitted by one or more cameras in the nursing home according to the text information of the target object, so as to locate the target object in the video stream information and obtain the image information of the target object.

[0149] It should be noted that the above explanations of the method embodiment are also applicable to the device of this embodiment and will not be repeated here.

[0150] The present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, the above-mentioned multimodal-based visual language positioning method is implemented.

[0151] The present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the above-mentioned multimodal visual language localization method.

[0152] Figure 2 It is an exemplary structural diagram of an electronic device capable of implementing the multimodal visual language localization method provided according to an embodiment of the present application.

[0153] like Figure 2 As shown, the electronic device includes an input device 501, an input interface 502, a central processor 503, a memory 504, an output interface 505, and an output device 506. The input interface 502, the central processor 503, the memory 504, and the output interface 505 are interconnected through a bus 507, and the input device 501 and the output device 506 are connected to the bus 507 through the input interface 502 and the output interface 505, respectively, and then connected to other components of the electronic device. Specifically, the input device 504 receives input information from the outside, and transmits the input information to the central processor 503 through the input interface 502; the central processor 503 processes the input information based on the computer executable instructions stored in the memory 504 to generate output information, temporarily or permanently stores the output information in the memory 504, and then transmits the output information to the output device 506 through the output interface 505; the output device 506 outputs the output information to the outside of the electronic device for use by the user.

[0154] That is to say, Figure 2 The electronic device shown can also be implemented as comprising: a memory storing computer executable instructions; and one or more processors, which can implement the combination when executing the computer executable instructions. Figure 1 Described in the paper is a multimodal visual-language localization method.

[0155] In one embodiment, Figure 2 The electronic device shown can be implemented to include: a memory 504 configured to store executable program codes; and one or more processors 503 configured to run the executable program codes stored in the memory 504 to execute the multimodal-based visual language localization method in the above-mentioned embodiment.

[0156] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0157] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0158] Computer readable media include permanent and non-permanent, removable and non-removable, and the media can be implemented by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), data versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0159] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0160] In addition, it is obvious that the word "comprising" does not exclude other units or steps. Multiple units, modules or devices stated in the device claim can also be implemented by one unit or the overall device through software or hardware.

[0161] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and a module, a program segment or a part of a code includes one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes identified in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or the total flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0162] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0163] In addition, it is obvious that the word "comprising" does not exclude other units or steps. Multiple units, modules or devices stated in the device claim can also be implemented by one unit or the overall device through software or hardware.

[0164] Although the present application is disclosed as above in terms of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

[0165] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application is described in detail with reference to the above embodiments, a person skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multimodal visual language localization method for interacting with the elderly in nursing homes, characterized in that: The multimodal visual language localization method includes: Acquire the person-searching instruction information obtained through interaction with the user; Acquire the head portrait information to be identified according to the person-searching instruction information; Obtain a custom gesture information library based on the avatar information to be recognized; Generate instruction information; Obtaining video stream information and voice information fed back by the user according to the instruction information; Acquire first text information according to the custom gesture information library and the video stream information; Acquire second text information according to the voice information; Get the trained semantic understanding model; Inputting the first text information and the second text information into a trained semantic understanding model to obtain target object text information; The video stream information transmitted by one or more camera devices in the nursing home is retrieved according to the target object text information, so as to locate the target object in the video stream information and obtain the image information of the target object.

2. The multimodal visual language localization method according to claim 1, characterized in that: The acquiring of the head portrait information to be identified according to the person-searching instruction information comprises: When the person-searching instruction information is obtained, image request information is generated and sent to the user; Obtaining the answer information fed back by the user according to the image request information; When the answer information is yes, a camera device of the human-machine interaction device generates a shooting signal; Acquire the head portrait information to be identified photographed by the camera device of the human-computer interaction device according to the shooting signal.

3. The multimodal visual language localization method according to claim 2, characterized in that: The step of acquiring a custom gesture information library according to the head portrait information to be recognized includes: Obtaining an avatar gesture library corresponding table, wherein the avatar gesture library corresponding table includes at least one preset avatar and a custom gesture information library corresponding to each preset avatar, wherein the custom gesture information library includes at least one preset custom gesture, and each preset custom gesture corresponds to a sentence information or word information; The similarity between the head portrait information to be identified and each preset head portrait is calculated respectively, so as to obtain a custom gesture information library corresponding to the preset head portraits whose similarity exceeds a preset threshold.

4. The multimodal visual language localization method according to claim 3, characterized in that: The acquiring the first text information according to the custom gesture information library and the video stream information comprises: Parsing the video stream information to obtain gesture information in the video stream information; Sentence information or word information corresponding to the preset custom gestures identical to each gesture information is respectively acquired from the custom gesture information library, and each of the sentence information or word information constitutes the first text information.

5. The multimodal visual language localization method according to claim 4, characterized in that: The trained semantic understanding model is a BERT model.

6. The multimodal visual language localization method according to claim 5, characterized in that: The step of retrieving the video stream information transmitted by one or more camera devices in the nursing home according to the target object text information, thereby locating the target object in the video stream information and obtaining the image information of the target object includes: Identify the target object text information, determine whether the camera position and time period information can be determined from the target object text information, and if so, Acquire a camera position database, wherein the camera position database includes at least one preset camera position and a camera corresponding to each preset camera position; Acquire a camera device corresponding to a preset camera device position that is the same as the camera device position, the camera device being the designated camera device; Obtaining video stream information of a specified camera device within a time period; The target object is located for each frame of image information in the video stream information according to the target object text information, and when the target object is located, the image information of the target object is obtained.

7. The multimodal visual language localization method according to claim 6, characterized in that: The step of retrieving the video stream information transmitted by one or more camera devices in the nursing home according to the target object text information, thereby locating the target object in the video stream information and obtaining the image information of the target object further includes: Identify the target object text information, determine whether the camera position and time period information can be determined from the target object text information, and if not, Determine whether the camera position can be determined from the target object text information, and if so, Acquire a camera position database, wherein the camera position database includes at least one preset camera position and a camera corresponding to each preset camera position; Acquire a camera device corresponding to a preset camera device position that is the same as the camera device position, the camera device being the designated camera device; Generate time period query information, wherein the time period query information includes multiple time period options; Obtaining a time period option selected by the user according to the time period inquiry information; Get the video stream information of the specified camera device within the time period option; The target object is located for each frame of image information in the video stream information according to the target object text information, and when the target object is located, the image information of the target object is obtained.

8. The multimodal visual language localization method according to claim 7, characterized in that: The step of retrieving the video stream information transmitted by one or more camera devices in the nursing home according to the target object text information, thereby locating the target object in the video stream information and obtaining the image information of the target object further includes: Identify the target object text information, determine whether the camera position and time period information can be determined from the target object text information, and if not, Determine whether the camera position can be determined from the target object text information, if not, then Determine whether the time period information can be determined from the target object text information. If so, Acquire video stream information of a camera device closest to the user within the time period; The target object is located for each frame of image information in the video stream information according to the target object text information, and when the target object is located, the image information of the target object is obtained.

9. The multimodal visual language localization method according to any one of claims 6 to 8, characterized in that: When the target object cannot be located in the acquired video stream information, the multimodal visual language localization method further includes: The adjacent camera devices to the camera device are selected in sequence through the adjacent selection method for identification until the target object is located.

10. A multimodal visual language localization device, characterized in that: The multimodal visual language localization device comprises: A person-searching instruction information acquisition module, the person-searching instruction information acquisition module is used to acquire person-searching instruction information acquired through interaction with a user; A module for acquiring head portrait information to be identified, the module for acquiring head portrait information to be identified is used to acquire head portrait information to be identified according to the person-searching instruction information; A custom gesture information library acquisition module, wherein the custom gesture information library acquisition module is used to acquire a custom gesture information library according to the avatar information to be identified; An indication information generating module, wherein the indication information generating module is used to generate indication information; A video stream information and voice information acquisition module, which is used to acquire the video stream information and voice information fed back by the user according to the instruction information; A first text information acquisition module, the first text information acquisition module is used to acquire first text information according to the custom gesture information library and the video stream information; A second text information acquisition module, the second text information acquisition module is used to acquire second text information according to the voice information; A semantic understanding model acquisition module, wherein the semantic understanding model acquisition module is used to acquire a trained semantic understanding model; A target object text information acquisition module, wherein the target object text information acquisition module is used to input the first text information and the second text information into a trained semantic understanding model, thereby acquiring the target object text information; A positioning module is used to retrieve the video stream information transmitted by one or more cameras in the nursing home according to the text information of the target object, so as to locate the target object in the video stream information and obtain the image information of the target object.

Images