Robot question and answer response generation method, device, computer device, readable storage medium and program product

Abstract

The present application relates to a method, apparatus, computer device, readable storage medium, and program product for generating a robot question-and-answer response. The method comprises: obtaining scene description information corresponding to the current scene, the scene description information being the attention processing result between any two elements obtained by performing attention mechanism processing on the target scene image; receiving query information; inputting the query information and the cached scene description information into a pre-trained model to obtain a response result corresponding to the query information, including: determining the attention result between the query information and each element in the target scene image through the pre-trained model, and obtaining a response result corresponding to the query information based on the attention processing result between any two elements and the attention result between the query information and each element in the target scene image. This method makes full use of the scene description information, reduces the amount of data processing for the model, and improves processing efficiency.

Description

Technical Field

[0001] The present application relates to the field of robot control technology, and in particular to a method, apparatus, computer device, readable storage medium, and program product for generating robot question and answer responses. Background Art

[0002] With the development of artificial intelligence technology, the application of robots in different scenarios has increased, especially in service, education, and medical scenarios. Every time a user enters a new query, the scenario and query are decoded and processed through large models such as GPT. Then, the answer is obtained and output based on the processing results. In this way, the information in the scenario corresponding to the large model needs to be decoded each time, which repeats the operation and reduces efficiency. Summary of the Invention

[0003] Based on this, it is necessary to provide a robot question and answer response generation method, device, computer equipment, computer-readable storage medium and computer program product that can reduce the data processing volume of the model and improve processing efficiency in response to the above technical problems.

[0004] In a first aspect, the present application provides a method for generating a robot question and answer response, the method comprising:

[0005] Obtain scene description information corresponding to the current scene, where the scene description information is an attention processing result between any two elements obtained by performing attention mechanism processing on the target scene image;

[0006] Receive inquiry information;

[0007] The query information and the cached scene description information are input into a pre-trained model to obtain a response result corresponding to the query information.

[0008] In one embodiment, obtaining scene description information corresponding to the current scene includes:

[0009] Acquire the target scene image;

[0010] Inputting the target scene image into a pre-trained model, and using the processing result of the attention mechanism in the pre-trained model as scene description information;

[0011] The scene description information is cached.

[0012] In one embodiment, acquiring the target scene image includes:

[0013] Determine the difference between the current scene image and the historical scene image;

[0014] When the difference is greater than a threshold, the current scene image is used as a target scene image.

[0015] In one embodiment, inputting the target scene image into a pre-trained model and using the processing result of the attention mechanism in the pre-trained model as scene description information includes:

[0016] Extract the scene sequence corresponding to the scene image through the feature extraction network of the pre-trained model;

[0017] Taking the scene sequence as the input of the decoder, and using the attention mechanism of the decoder to calculate the attention results between different elements in the scene sequence;

[0018] The attention result is used as scene description information.

[0019] In one embodiment, caching the scene description information includes:

[0020] The scene description information is stored in a k-value cache, where the index k of the scene description information in the k-value cache is an element, and the value value is the attention result between any two elements.

[0021] In one embodiment, inputting the query information and the cached scene description information into a pre-trained model to obtain a response result corresponding to the query information includes:

[0022] If the response result is that no answer corresponding to the inquiry information is obtained, performing keyword extraction on the inquiry information;

[0023] Performing image segmentation on the target scene image based on the extracted keywords to obtain a region of interest;

[0024] The query information and the region of interest are input into a pre-trained model for processing to obtain a target response result.

[0025] In a second aspect, the present application further provides a robot question-and-answer response generating device, the device comprising:

[0026] A scene description information acquisition module is used to obtain scene description information corresponding to the current scene, wherein the scene description information is an attention processing result between any two elements obtained by performing an attention mechanism processing on the target scene image;

[0027] A receiving module, used for receiving inquiry information;

[0028] The response module is used to input the query information and the cached scene description information into a pre-trained model to obtain a response result corresponding to the query information.

[0029] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in any one of the above embodiments when executing the computer program.

[0030] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the method in any one of the above-mentioned embodiments when the computer program is executed by a processor.

[0031] In a fifth aspect, the present application also provides a computer program product, comprising a computer program, which implements the steps of the method in any one of the above embodiments when executed by a processor.

[0032] The above-mentioned robot question and answer response generation method, device, computer equipment, readable storage medium and program product first obtain the scene image, and obtain scene description information based on the scene image, and cache it in advance. When making a subsequent inquiry, it is directly processed based on the inquiry information and the scene description information without having to recalculate the attention result between the scene description information, thereby making full use of the scene description information, reducing the data processing amount of the model, and improving processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0034] Figure 1 1 is a flow chart of a method for generating a robot question and answer response in one embodiment;

[0035] Figure 2 A flowchart of the steps for obtaining scene description information in one embodiment;

[0036] Figure 3 FIG. 1 is a flowchart of a process in an embodiment in which a response result is that no answer corresponding to the query information is obtained;

[0037] Figure 4 is a structural block diagram of a robot question and answer response generating device in one embodiment;

[0038] Figure 5FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0040] In one embodiment, Figure 1 As shown, a method for generating a robot question and answer response is provided. This embodiment uses the method applied to a robot terminal as an example for illustration. It is understandable that the method can also be applied to a server corresponding to the robot terminal, and can also be applied to a system including a robot terminal and a server, and is implemented through the interaction between the robot terminal and the server. In this embodiment, the method includes the following steps:

[0041] S102: Obtain scene description information corresponding to the current scene, where the scene description information is an attention processing result between any two elements obtained by performing attention mechanism processing on the target scene image.

[0042] The scene description information is obtained by performing attention mechanism processing based on the target scene image, and the scene description information is used to describe the attention processing result between any two element tokens in the scene. The target scene image is an image of the scene in which the robot is located, and the robot performs image acquisition in real time during operation. In some optional embodiments, the robot may include multiple cameras, each of which can capture a panoramic image of the scene in which the robot is located. In other embodiments, the robot may include only one camera or a binocular camera, and the field of view of the camera is the field of view of the robot, so that the image captured by the camera can represent the image of the scene in which the robot is located.

[0043] The attention mechanism is the attention mechanism of the decoder in the pre-trained model VLM, which is used to calculate the attention processing results between any two element tokens output by the feature extraction network.

[0044] S104: Receive inquiry information.

[0045] The inquiry information is information input by the user, for example, the inquiry information may be “what is included in the scene”, “what are the details of an object in the scene”, etc. The user can make inquiries based on needs.

[0046] After receiving the query information, the query information is processed to obtain a text sequence, for example, the text is segmented to obtain a vector corresponding to each segmented word, and then the vectors are connected to obtain a text sequence corresponding to the query information. In other embodiments, other processing methods can also be used.

[0047] In addition, it should be noted that if the inquiry information is in the form of voice, in this application, the inquiry information is first converted from voice to text to obtain text information, and then the text information is processed.

[0048] S106: Input the query information and the cached scene description information into a pre-trained model to obtain a response result corresponding to the query information, including: determining the attention result between the query information and each element in the target scene image through a pre-trained model, and obtaining a response result corresponding to the query information based on the attention processing result between the any two elements and the attention result between the query information and each element in the target scene image.

[0049] The pre-trained model VLM includes an attention mechanism, which needs to calculate the attention results between any two element tokens. In this application, it is assumed that the scene description information includes the attention results between three element tokens, namely A, B and C. The scene description information includes the attention results between AB, the attention results between AC and the attention results between BC. Now there is query information. If you want to get the corresponding image-text conversion results, you also need to calculate the attention results between the query information Q and the above three element tokens, that is, the attention results between AQ, the attention results between BQ and the attention results between CQ.

[0050] In traditional technology, each time a query is received, the scene image and query need to be input into the VLM. First, feature extraction is performed on the scene image, and text extraction is performed on the query. Then, the attention mechanism is calculated through transformation. Taking the above example, it is necessary to calculate the attention results between AB, the attention results between AC, the attention results between AQ, the attention results between BC, the attention results between BQ, and the attention results between CQ. These attention results need to be calculated each time a query is input, resulting in a large amount of repeated calculations.

[0051] To address the aforementioned technical issues, this application stores the attention results between AB, AC, and BC as scene description information. This allows subsequent processing to calculate only the attention results between each element token of the scene image and the query information Q. In this application, each element token of the scene image is also cached. This reduces the repeated calculation of scene description information involved in the query processing, improving processing efficiency.

[0052] The above-mentioned robot question-answer response generation method first obtains the scene image, and obtains the scene description information based on the scene image, and caches it in advance. When making subsequent inquiries, it is directly processed based on the query information and the scene description information without having to recalculate the attention results between the scene description information. This makes full use of the scene description information, reduces the data processing amount of the model, and improves processing efficiency.

[0053] In one of the optional embodiments, combined with Figure 2 As shown, Figure 2 This is a flowchart of a scene description information acquisition step in an embodiment. The scene description information acquisition step, i.e., acquiring scene description information corresponding to the current scene, includes: acquiring a target scene image; inputting the target scene image into a pre-trained model, and using the processing result of the attention mechanism in the pre-trained model as scene description information; and caching the scene description information.

[0054] The pre-trained model VLM is a model that includes robot vision and a large prediction model. The structure of the large prediction model can be a transform structure, so that the transform's attention mechanism can be used, such as the cross-attention mechanism, to calculate the cross-attention results of each element in the scene.

[0055] The pre-trained model VLM includes a feature extraction network and a decoder corresponding to the transform. The feature extraction network is used to extract the scene sequence corresponding to the scene image, and then input the scene sequence into the transform for decoding. The decoding process of the transform includes the processing of the cross-attention mechanism, which can calculate the attention results between different elements in the scene sequence and cache the attention results as scene description information.

[0056] In the above embodiment, scene description information is generated in each new scene, so when the query is subsequently processed, there is no need to process the scene image, but the scene description information and the query are directly processed, which reduces the repeated calculation of the scene description information and improves the calculation efficiency.

[0057] In one optional embodiment, acquiring the target scene image includes: determining a difference between a current scene image and a historical scene image; and using the current scene image as the target scene image when the difference is greater than a threshold.

[0058] The current scene image is a scene image captured in real time by the robot's camera, and the historical scene image is a scene image previously captured by the robot's camera. The historical scene image can be the target scene image of the previous scene, or any scene image captured from the previous scene to the current time. Optionally, the present application uses the historical scene image as the target scene image of the previous scene as an example for explanation.

[0059] Calculating the difference between the current scene image and the historical scene image can be performed based on SLAM. Specifically, calculating the difference between the current scene image and the historical scene image can be performed by calculating the difference in feature points between the current scene image and the historical scene image. For example, the common parts in the current scene image and the historical scene image can be calculated, and then the ratio of the number of pixels in the common parts to the number of pixels in the current scene image can be used as the difference. In other embodiments, the difference can also be determined based on optical flow information, fixed objects, etc., which are not specifically limited here.

[0060] The threshold is preset and can be obtained based on experience, such as 75%. In other embodiments, other values can also be selected and are not specifically limited here.

[0061] When the difference is greater than the threshold, it is determined that the scene has changed, and it is necessary to obtain the target scene image corresponding to the scene. Optionally, in this application, for ease of processing, the current scene image with a difference greater than the threshold is directly used as the target scene image, that is, the first frame scene image entering the new scene.

[0062] In the above embodiment, the scene change can be judged based on the context scene image, thereby ensuring the real-time performance of the target scene image and further ensuring the accuracy of the scene description information.

[0063] In one of the optional embodiments, the target scene image is input into a pre-trained model, and the processing result of the attention mechanism in the pre-trained model is used as the scene description information, including: extracting the scene sequence corresponding to the scene image through the feature extraction network of the pre-trained model; using the scene sequence as the input of the decoder, and using the attention mechanism of the decoder to calculate the attention result between different elements in the scene sequence; and using the attention result as the scene description information.

[0064] Among them, the feature extraction network can be a simple neural network, which can extract the scene sequence corresponding to the scene image. Optionally, the feature extraction network can be connected to a fully connected layer. After the scene sequence is extracted by the scene extraction network, it is then classified through the fully connected layer to complete the prediction of each element token, and then the obtained element token is input into the decoder as a scene sequence.

[0065] The decoder is transform, which may include an attention mechanism. The attention mechanism may include a cross-attention mechanism, through which the attention result between any two element tokens can be calculated.

[0066] In the above embodiment, after entering a new scene, the scene description information is first calculated as the basis for subsequent inquiries. In this way, there is no need to calculate the attention results between any two element tokens in the scene during subsequent inquiries, which can improve processing efficiency.

[0067] In one of the optional embodiments, caching the scene description information includes: storing the scene description information in a k-value cache, wherein the index k of the scene description information in the k-value cache is an element, and the value value is the attention result between any two elements.

[0068] The scene description information includes the attention results between different element tokens, where the index can be an element token and the value is the attention result between any two element tokens.

[0069] In one of the optional embodiments, combined with Figure 3 As shown, Figure 3 This is a processing flow chart for an embodiment in which a response result is that no answer corresponding to the query information is obtained, wherein the query information and the cached scene description information are input into a pre-trained model to obtain a response result corresponding to the query information, including: when the response result is that no answer corresponding to the query information is obtained, keyword extraction is performed on the query information; image segmentation is performed on the target scene image based on the extracted keywords to obtain a region of interest; and the query information and the region of interest are input into a pre-trained model for processing to obtain a target response result.

[0070] Among them, the response result may include the answer corresponding to the inquiry information, and the answer corresponding to the inquiry information that cannot be obtained. If the response result is the answer corresponding to the inquiry information, the response result can be directly output. If the response result is that the answer corresponding to the inquiry information cannot be obtained, one way is to output the answer that cannot be obtained. Another way is to generate response information in combination with context information when the response result is that the answer corresponding to the inquiry information cannot be obtained.

[0071] The keyword extraction of the query information can be performed through a large prediction model. After obtaining the target scene image, the target scene image is processed through the VLM, and the scene elements, that is, the element information corresponding to the element token, such as table, chair, etc. can also be output.

[0072] By using the big prediction model to extract keywords from query information, keywords can be extracted from the query based on the scene elements output by the VLM, such as the keyword "table".

[0073] The target scene image is segmented based on the extracted keywords. This can be done by inputting the keywords and the target scene image into a segmentation model. The segmentation model can be a SAM model, etc., which is not specifically limited here. The target scene image is segmented based on the keywords by the segmentation model to determine a region of interest corresponding to the query information, and then the region of interest is segmented from the target scene image. This can reduce the amount of image processing.

[0074] Finally, the region of interest and query information are input into the VLM for processing, where the feature extraction network of the VLM is used to extract the image sequence of the region of interest, and the text processing module is used to extract the text features of the query information to obtain a text sequence. The image sequence and text sequence are then input into the VLM decoder, and the attention mechanism of the VLM decoder is used to perform attention processing on the image sequence and text sequence, and finally the target response result is obtained.

[0075] In the above embodiment, when no answer corresponding to the query information is obtained, the contextual query information can be used to segment the scene image to obtain the region of interest, and then VLM processing can be performed based on the query information and the scene image to obtain the target response result, which makes full use of the contextual information and makes the robot response result more accurate.

[0076] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0077] Based on the same inventive concept, embodiments of the present application also provide a robot question and answer response generation device for implementing the aforementioned robot question and answer response generation method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more robot question and answer response generation device embodiments provided below can be found in the above-mentioned limitations of the robot question and answer response generation method, and will not be further elaborated here.

[0078] In an exemplary embodiment, Figure 4 As shown, a robot question-answer response generating device is provided, comprising: a scene description information acquisition module 401, a receiving module 402 and a response module 403, wherein:

[0079] A scene description information acquisition module 401 is used to obtain scene description information corresponding to the current scene, wherein the scene description information is an attention processing result between any two elements obtained by performing an attention mechanism processing on the target scene image;

[0080] Receiving module 402, for receiving inquiry information;

[0081] The response module 403 is used to input the query information and the cached scene description information into a pre-trained model to obtain a response result corresponding to the query information, including: determining the attention result between the query information and each element in the target scene image through a pre-trained model, and obtaining a response result corresponding to the query information based on the attention processing result between any two elements and the attention result between the query information and each element in the target scene image.

[0082] In one of the optional embodiments, the above-mentioned scene description information acquisition module 401 is specifically used to obtain a target scene image; input the target scene image into a pre-trained model, and use the processing result of the attention mechanism in the pre-trained model as the scene description information; and cache the scene description information.

[0083] In one optional embodiment, the scene description information acquisition module 401 is specifically used to determine the difference between the current scene image and the historical scene image; when the difference is greater than a threshold, the current scene image is used as the target scene image.

[0084] In one of the optional embodiments, the above-mentioned scene description information acquisition module 401 is specifically used to extract the scene sequence corresponding to the scene image through the feature extraction network of a pre-trained model; use the scene sequence as the input of the decoder, and use the attention mechanism of the decoder to calculate the attention results between different elements in the scene sequence; and use the attention results as scene description information.

[0085] In one of the optional embodiments, the above-mentioned scene description information acquisition module 401 is specifically used to store the scene description information in a k-value cache, wherein the index k of the scene description information in the k-value cache is an element, and the value value is the attention result between any two elements.

[0086] In one of the optional embodiments, the above-mentioned response module 403 is specifically used to extract keywords from the query information when the response result is that no answer corresponding to the query information is obtained; perform image segmentation on the target scene image based on the extracted keywords to obtain a region of interest; and input the query information and the region of interest into a pre-trained model for processing to obtain a target response result.

[0087] Each module in the aforementioned robot question-and-answer response generation device may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in hardware form, or may be stored in a computer device memory in software form, so that the processor can call and execute the corresponding operations of each module.

[0088] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 5As shown. The computer device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals via wired or wireless means, and the wireless means can be implemented via Wi-Fi, a mobile cellular network, near-field communication (NFC), or other technologies. When executed by the processor, the computer program implements a method for generating robot question-and-answer responses. The display unit of the computer device is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse.

[0089] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0090] In an exemplary embodiment, a computer device is provided, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the following steps when executing the computer program: obtaining scene description information corresponding to a current scene, wherein the scene description information is an attention processing result between any two elements obtained by performing attention mechanism processing on a target scene image; receiving query information; inputting the query information and the cached scene description information into a pre-trained model to obtain a response result corresponding to the query information, including: determining the attention result between the query information and each element in the target scene image through a pre-trained model, and obtaining a response result corresponding to the query information based on the attention processing result between the any two elements and the attention result between the query information and each element in the target scene image.

[0091] In one embodiment, the obtaining of scene description information corresponding to the current scene implemented when the processor executes a computer program includes: obtaining a target scene image; inputting the target scene image into a pre-trained model, and using the processing result of the attention mechanism in the pre-trained model as the scene description information; and caching the scene description information.

[0092] In one embodiment, the acquiring of the target scene image implemented when the processor executes the computer program includes: determining the difference between the current scene image and the historical scene image; and using the current scene image as the target scene image when the difference is greater than a threshold.

[0093] In one embodiment, the inputting of the target scene image into a pre-trained model and the use of the processing results of the attention mechanism in the pre-trained model as scene description information implemented by the processor when executing a computer program include: extracting a scene sequence corresponding to the scene image through a feature extraction network of the pre-trained model; using the scene sequence as input to a decoder, and using the attention mechanism of the decoder to calculate the attention results between different elements in the scene sequence; and using the attention results as scene description information.

[0094] In one embodiment, the caching of the scene description information implemented when the processor executes a computer program includes: storing the scene description information in a k-value cache, wherein the index k of the scene description information in the k-value cache is an element, and the value value is the attention result between any two elements.

[0095] In one embodiment, the step of inputting the query information and the cached scene description information into a pre-trained model to obtain a response result corresponding to the query information implemented when a processor executes a computer program includes: when the response result is that no answer corresponding to the query information is obtained, performing keyword extraction on the query information; performing image segmentation on the target scene image based on the extracted keywords to obtain a region of interest; and inputting the query information and the region of interest into a pre-trained model for processing to obtain a target response result.

[0096] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: obtaining scene description information corresponding to the current scene, the scene description information being an attention processing result between any two elements obtained by performing attention mechanism processing on a target scene image; receiving query information; inputting the query information and the cached scene description information into a pre-trained model to obtain a response result corresponding to the query information, including: determining the attention result between the query information and each element in the target scene image through a pre-trained model, and obtaining a response result corresponding to the query information based on the attention processing result between the any two elements and the attention result between the query information and each element in the target scene image.

[0097] In one embodiment, the method of obtaining scene description information corresponding to the current scene implemented when the computer program is executed by the processor includes: obtaining a target scene image; inputting the target scene image into a pre-trained model, and using the processing result of the attention mechanism in the pre-trained model as the scene description information; and caching the scene description information.

[0098] In one embodiment, the acquiring of the target scene image implemented when the computer program is executed by the processor includes: determining the difference between the current scene image and the historical scene image; and using the current scene image as the target scene image when the difference is greater than a threshold.

[0099] In one embodiment, the method implemented when a computer program is executed by a processor of inputting the target scene image into a pre-trained model and using the processing result of the attention mechanism in the pre-trained model as scene description information includes: extracting a scene sequence corresponding to the scene image through a feature extraction network of the pre-trained model; using the scene sequence as input to a decoder, and using the attention mechanism of the decoder to calculate the attention result between different elements in the scene sequence; and using the attention result as the scene description information.

[0100] In one embodiment, the caching of the scene description information implemented when the computer program is executed by the processor includes: storing the scene description information in a k-value cache, wherein the index k of the scene description information in the k-value cache is an element, and the value value is the attention result between any two elements.

[0101] In one embodiment, the computer program implemented when executed by a processor inputs the query information and the cached scene description information into a pre-trained model to obtain a response result corresponding to the query information, including: when the response result is that no answer corresponding to the query information is obtained, performing keyword extraction on the query information; performing image segmentation on the target scene image based on the extracted keywords to obtain a region of interest; and inputting the query information and the region of interest into a pre-trained model for processing to obtain a target response result.

[0102] In one embodiment, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the following steps: obtaining scene description information corresponding to a current scene, the scene description information being an attention processing result between any two elements obtained by performing attention mechanism processing on a target scene image; receiving query information; inputting the query information and the cached scene description information into a pre-trained model to obtain a response result corresponding to the query information, comprising: determining the attention result between the query information and each element in the target scene image through a pre-trained model, and obtaining a response result corresponding to the query information based on the attention processing result between the any two elements and the attention result between the query information and each element in the target scene image.

[0103] In one embodiment, the method of obtaining scene description information corresponding to the current scene implemented when the computer program is executed by the processor includes: obtaining a target scene image; inputting the target scene image into a pre-trained model, and using the processing result of the attention mechanism in the pre-trained model as the scene description information; and caching the scene description information.

[0104] In one embodiment, the acquiring of the target scene image implemented when the computer program is executed by the processor includes: determining the difference between the current scene image and the historical scene image; and using the current scene image as the target scene image when the difference is greater than a threshold.

[0105] In one embodiment, the method implemented when a computer program is executed by a processor of inputting the target scene image into a pre-trained model and using the processing result of the attention mechanism in the pre-trained model as scene description information includes: extracting a scene sequence corresponding to the scene image through a feature extraction network of the pre-trained model; using the scene sequence as input to a decoder, and using the attention mechanism of the decoder to calculate the attention result between different elements in the scene sequence; and using the attention result as the scene description information.

[0106] In one embodiment, the caching of the scene description information implemented when the computer program is executed by the processor includes: storing the scene description information in a k-value cache, wherein the index k of the scene description information in the k-value cache is an element, and the value value is the attention result between any two elements.

[0107] In one embodiment, the computer program implemented when executed by a processor inputs the query information and the cached scene description information into a pre-trained model to obtain a response result corresponding to the query information, including: when the response result is that no answer corresponding to the query information is obtained, performing keyword extraction on the query information; performing image segmentation on the target scene image based on the extracted keywords to obtain a region of interest; and inputting the query information and the region of interest into a pre-trained model for processing to obtain a target response result.

[0108] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0109] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), quantum computing-based data processing logic devices, artificial intelligence (AI) processors, and the like.

[0110] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0111] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A robot question and answer response generation method, characterized in that: The method comprises: Obtain scene description information corresponding to the current scene, where the scene description information is an attention processing result between any two elements obtained by performing attention mechanism processing on the target scene image; Receive inquiry information; Inputting the query information and the cached scene description information into a pre-trained model to obtain a response result corresponding to the query information, including: determining an attention result between the query information and each element in the target scene image through the pre-trained model, and obtaining a response result corresponding to the query information based on an attention processing result between the arbitrary two elements and an attention result between the query information and each element in the target scene image; The step of inputting the query information and the cached scene description information into a pre-trained model to obtain a response result corresponding to the query information includes: If the response result is that no answer corresponding to the inquiry information is obtained, performing keyword extraction on the inquiry information; Performing image segmentation on the target scene image based on the extracted keywords to obtain a region of interest; The query information and the region of interest are input into a pre-trained model for processing to obtain a target response result.

2. The method according to claim 1, characterized in that The obtaining of scene description information corresponding to the current scene includes: Acquire the target scene image; Inputting the target scene image into a pre-trained model, and using the processing result of the attention mechanism in the pre-trained model as scene description information; The scene description information is cached.

3. The method according to claim 2, characterized in that The acquiring of the target scene image comprises: Determine the difference between the current scene image and the historical scene image; When the difference is greater than a threshold, the current scene image is used as a target scene image.

4. The method according to claim 2, characterized in that The step of inputting the target scene image into a pre-trained model and using the processing result of the attention mechanism in the pre-trained model as scene description information includes: Extract the scene sequence corresponding to the scene image through the feature extraction network of the pre-trained model; Taking the scene sequence as the input of the decoder, and using the attention mechanism of the decoder to calculate the attention results between different elements in the scene sequence; The attention result is used as scene description information.

5. The method according to claim 2, characterized in that The caching of the scene description information includes: The scene description information is stored in a k-value cache, where the index k of the scene description information in the k-value cache is an element, and the value value is the attention result between any two elements.

6. A robot question and answer response generating device, characterized in that: The device comprises: A scene description information acquisition module is used to obtain scene description information corresponding to the current scene, wherein the scene description information is an attention processing result between any two elements obtained by performing an attention mechanism processing on the target scene image; A receiving module, used for receiving inquiry information; a response module, configured to input the query information and the cached scene description information into a pre-trained model to obtain a response result corresponding to the query information, including: determining an attention result between the query information and each element in the target scene image through the pre-trained model, and obtaining a response result corresponding to the query information based on an attention processing result between the arbitrary two elements and an attention result between the query information and each element in the target scene image; The response module is specifically used to extract keywords from the query information when the response result is that no answer corresponding to the query information is obtained; perform image segmentation on the target scene image based on the extracted keywords to obtain a region of interest; and input the query information and the region of interest into a pre-trained model for processing to obtain a target response result.

7. The device according to claim 6, characterized in that The scene description information acquisition module is specifically used to obtain a target scene image; input the target scene image into a pre-trained model, and use the processing result of the attention mechanism in the pre-trained model as the scene description information; The scene description information is cached.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images