Image-based human-machine interaction method, device, equipment, and storage medium

Abstract

To provide an image-based man-machine interaction method and device, instrument and a storage medium which can improve accuracy of question-and-answer by extracting layout information and content information from an image and improve a man-machine interaction experience of a user.SOLUTION: According to a method, an image to be analyzed is acquired, and image layout information and image content information of the image to be analyzed are determined; wherein the image to be analyzed comprises various types of modal data, the layout information represents the distribution condition of the image elements with the preset granularity in the image to be analyzed, and the content information represents the content expressed by the modal data in the image to be analyzed; in response to the acquired question information, the method determines response information corresponding to the question information according to the image layout information and the image content information; wherein the question information represents a question proposed by the user for the image to be analyzed, and the response information represents a reply answer corresponding to the question information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to the field of image processing in the field of artificial intelligence, and in particular to an image-based human-machine interaction method, device, apparatus, and storage medium. [Background technology]

[0002] With the development of the field of artificial intelligence, automatic image understanding and analysis technology is gradually being applied in various industries, bringing convenience to people's work. For example, users can ask questions about images and get automatic answers based on the content in the image.

[0003] However, the types of pictures used in work situations are complex, including, for example, document screenshots, table screenshots, flowcharts, architecture diagrams, etc. Current picture understanding technologies have poor ability to understand the content of these pictures, which fails to meet user needs and results in a poor human-machine interaction experience. Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure provides an image-based human-machine interaction method, apparatus, device, and storage medium. [Means for solving the problem]

[0005] According to a first aspect, the present disclosure provides an image-based human-machine interaction method, the method comprising: acquiring an image to be analyzed, the image including at least two types of modality data; a step of determining image layout information and image content information of the image to be analyzed, wherein the image layout information indicates a distribution state of image elements of a predetermined granularity in the image to be analyzed, and the image content information indicates content expressed by modality data in the image to be analyzed; and a step of determining response information corresponding to the question information in accordance with the image layout information and the image content information in response to the acquired question information, wherein the question information indicates a question posed about the image to be analyzed, and the response information indicates a reply answer corresponding to the question information.

[0006] According to a second aspect, the present disclosure provides an image-based human-machine interaction device, the device comprising: an acquisition unit adapted to acquire an image to be analyzed, the image comprising at least two types of modality data; a first determination unit used to determine image layout information and image content information of the image to be analyzed, the image layout information indicating a distribution state of image elements of a predetermined granularity in the image to be analyzed, and the image content information indicating content expressed by modality data in the image to be analyzed; and a second determination unit used to respond to the acquired question information and determine response information corresponding to the question information according to the image layout information and the image content information, wherein the question information indicates a question posed about the image to be analyzed, and the response information indicates a reply answer corresponding to the question information.

[0007] According to a third aspect, the present disclosure provides an electronic device, the electronic device comprising: at least one processor; a memory communicatively coupled to the at least one processor; The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of the first aspect.

[0008] According to a fourth aspect, the present disclosure provides a non-transitory computer-readable storage medium having stored thereon computer instructions, the computer instructions being used to cause the computer to perform the method of the first aspect.

[0009] According to a fifth aspect, the present disclosure provides a computer program, which, when executed by a processor, implements the method according to the first aspect. [Effects of the Invention]

[0010] The technology disclosed herein improves the accuracy of image understanding and enhances the user's human-machine interaction experience.

[0011] It should be noted that the contents described in this section are not intended to identify key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will be readily apparent from the following detailed description. [Brief explanation of the drawings]

[0012] The drawings are used for better understanding of the present technical solution and are not intended to limit the present disclosure. [Figure 1] 1 is a flowchart of an image-based human-machine interaction method provided according to an embodiment of the present disclosure. [Figure 2] 1 is a flowchart of an image-based human-machine interaction method provided according to an embodiment of the present disclosure. [Figure 3] FIG. 1 is a schematic diagram of an image to be analyzed according to an embodiment of the present disclosure. [Figure 4]1 is a flowchart of an image-based human-machine interaction method provided according to an embodiment of the present disclosure. [Figure 5] FIG. 1 is a procedural diagram of human-machine interaction according to an embodiment of the present disclosure. [Figure 6] FIG. 1 is a structural block diagram of an image-based human-machine interaction device provided according to an embodiment of the present disclosure; [Figure 7] FIG. 1 is a structural block diagram of an image-based human-machine interaction device provided according to an embodiment of the present disclosure; [Figure 8] FIG. 1 is a block diagram of an electronic device for implementing an image-based human-machine interaction method according to an embodiment of the present disclosure. [Figure 9] FIG. 1 is a block diagram of an electronic device for implementing an image-based human-machine interaction method according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0013]

[0023] Hereinafter, exemplary embodiments of the present disclosure will be described with reference to the drawings. For ease of understanding, various details of the embodiments of the present disclosure are included therein and should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described in the detailed description without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the following description will omit descriptions of well-known functions and structures.

[0014] Users can ask questions about an image. For example, if a user wants an explanation of the image's content, they can request the generation of a memo describing the image's content. Current image processing technology makes it possible to recognize pictorial content in images. For example, models such as BLIP (Bootstrapping Language-Image Pre-training) can be used for image processing. However, currently, these models can only analyze and understand pure pictures, and are not very capable of understanding complex types of images. For example, they can recognize and analyze landscape photos, portrait photos, etc.

[0015] Compared with the types of pictures commonly seen in daily life, the types of pictures in work situations are more heterogeneous and complex, including pure pictures, document screenshots, table screenshots, flowcharts, architecture diagrams, data charts, etc. Traditional models cannot accurately understand the content of these pictures, which makes it easy to make mistakes in answers to user questions and results in a poor user experience of human-machine interaction.

[0016] The present disclosure provides an image-based human-machine interaction method, device, equipment, and storage medium that are applied to the image processing field in the field of artificial intelligence to improve the accuracy of image understanding and enhance user experience.

[0017] Note that the model in this example is not a model for a specific user and does not reflect the personal information of a specific user. Note that the images in this example are from the disclosed dataset.

[0018] The collection, storage, use, processing, transmission, provision and disclosure of users' personal information in connection with the technical solutions disclosed herein shall all comply with relevant laws and regulations and be handled in a manner that does not violate public order and morals.

[0019] To help the reader better understand the implementation principles of the present disclosure, the following examples will be described in more detail with reference to FIGS. 1 to 9.

[0020] 1 is a flowchart of an image-based human-machine interaction method provided according to an embodiment of the present disclosure, which can be performed by an image-based human-machine interaction device. As shown in FIG. 1, the method includes steps S101 to S103.

[0021] In S101, an image to be analyzed containing at least two types of modality data is acquired.

[0022] For example, when a user engages in human-machine interaction, the user can ask questions about the image, for example, if there are multiple people in the image, the user can ask how many people are in the image, or if the image depicts a traffic accident, the user can ask to explain the content of the image. The image that the user is targeting during human-machine interaction is set as the analysis target image, and the user's questions can be answered by understanding and analyzing the analysis target image.

[0023] A user can manually input or select the image to be analyzed. For example, when performing human-machine interaction, a user may first upload the image to be analyzed and then ask a question about the image to be analyzed. The image to be analyzed may contain multiple modality data, such as text modality and visual modality. Text modality refers to the inclusion of text in the image to be analyzed, and visual modality refers to the inclusion of a pattern in the image to be analyzed. For example, if the image to be analyzed is a screenshot of a webpage and the webpage displays text and scenery, the included text will be the text modality and the scenery will be the visual modality. For example, if the image to be analyzed is a flowchart, the multiple flow boxes included in the flowchart will be the visual modality, and the text included in the flow boxes will be the text modality. In this embodiment, the image to be analyzed may be of various image types, such as a photograph, a document screenshot, a table screenshot, a flowchart, an architecture diagram, or a data chart.

[0024] In S102, image layout information and image content information of the image to be analyzed are determined, where the image layout information indicates the distribution status of image elements of a predetermined granularity in the image to be analyzed, and the image content information indicates the content expressed by the modality data in the image to be analyzed.

[0025] For example, the system acquires an analysis target image confirmed by a user, and after obtaining the analysis target image, interprets and analyzes the analysis target image to determine image layout information and image content information of the analysis target image. Image layout information refers to the distribution of image elements of a predetermined granularity in the analysis target image, and image content information refers to the content expressed by modality data in the analysis target image. Image elements are components of the analysis target image. For example, image elements are characters in a picture. The predetermined granularity is a predetermined division size of the image elements. For example, if one word is the predetermined granularity, the image layout information is the distribution of each word in the analysis target image. The distribution can represent the position of image elements of the predetermined granularity in the analysis target image, and the position can be represented using coordinates, for example. That is, the analysis target image is divided according to the predetermined granularity to obtain image elements of the predetermined granularity, and the coordinates of each image element of the predetermined granularity in the analysis target image are determined, thereby obtaining image layout information of the analysis target image.

[0026] The image to be analyzed includes multiple types of modality data, and for each type of modality data, the content expressed by the modality data can be determined as content information for the modality data. That is, multiple content information can be obtained for the image to be analyzed. For example, in the case of a visual modality, an interpretation of the image content expressed by the visual modality can be obtained as content information, and in the case of a text modality, the text content of the text modality can be obtained as content information.

[0027] In S103, in response to the acquired question information, response information corresponding to the question information is determined according to the image layout information and image content information, where the question information indicates a question posed about the image to be analyzed, and the response information indicates a reply answer corresponding to the question information.

[0028] For example, a user can issue question information indicating a question the user has about the image to be analyzed, i.e., the user can ask a question about the image to be analyzed, and through human-machine interaction, an answer to the question posed by the user can be automatically obtained. The user's question information is acquired, and response information corresponding to the question information is determined according to the image layout information and image content information, and the response information is a reply to the question indicated by the question information.

[0029] A semantic analysis is performed on the question information, and information corresponding to the semantics of the question information can be extracted as response information from the image layout information and image content information. For example, if the user's question is "How many words are in the image?", the number of words in the image can be determined according to the image layout information. Alternatively, if the user's question is "What content does the image screen represent?", content information corresponding to the visual modality can be determined as response information. Algorithms such as semantic analysis can be preset for image processing and analysis, and in this embodiment, the preset algorithm used during image processing is not particularly limited.

[0030] The image to be analyzed may be input into a preset large-scale model, the image may be processed and analyzed by the large-scale model, and response information may be output from the large-scale model. The image processing and analysis may be completed by calling tools implemented by various means, such as deep learning, machine learning, and expert rules. In this embodiment, the tools used are not particularly limited.

[0031] In an embodiment of the present disclosure, the image to be analyzed may include data from multiple modalities, such as text and visual modalities. By acquiring the image to be analyzed, image layout information and image content information of the image to be analyzed are determined. The image layout information can indicate the distribution of image elements of a predetermined granularity in the image to be analyzed, and the image content information can indicate the content expressed by the modality data in the image to be analyzed. By determining the image layout information and image content information, the image to be analyzed can be comprehensively understood and analyzed, improving the comprehensiveness of information extraction and comprehension ability. In response to a user's question, response information corresponding to the question is determined according to the image layout information and image content information. This solves the problem of the inability of conventional technologies to understand complex multi-modality images, improves the accuracy of question answering, and enhances the human-machine interaction experience.

[0032] FIG. 2 is a flowchart of an image-based human-machine interaction method provided by an embodiment of the present disclosure.

[0033] In this embodiment, the step of determining image layout information of the image to be analyzed may be detailed as a step of determining image elements of a predetermined granularity in the image to be analyzed, where the image elements indicate the components of the image to be analyzed, and a step of determining the coordinate positions in the image to be analyzed of the image elements of a predetermined granularity, and determining the image layout information of the image to be analyzed according to the coordinate positions.

[0034] The modality data in the image to be analyzed includes a text modality and a visual modality, and the step of determining image content information of the image to be analyzed may be detailed as a step of performing a text extraction process of the text modality on the image to be analyzed to obtain first content information corresponding to the text modality, and a step of converting content expressed by the visual modality in the image to be analyzed into text written in a natural language to obtain second content information corresponding to the visual modality, wherein the image content information includes the first content information and the second content information.

[0035] As shown in FIG. 2, the method includes S201 to S203.

[0036] In S201, the image to be analyzed is acquired, and image elements of a predetermined granularity in the image to be analyzed are determined. The image elements indicate the components of the image to be analyzed, and the coordinate positions of the image elements of the predetermined granularity in the image to be analyzed are determined, and image layout information is determined according to the coordinate positions.

[0037] For example, after the image to be analyzed is acquired, image layout information and image content information of the image to be analyzed may be determined. In this embodiment, the order in which the image layout information and image content information are determined is not particularly limited. The predetermined granularity of the image elements is used to divide the image elements. When determining the image layout information, image elements of a predetermined granularity can be determined from the image to be analyzed, thereby determining multiple image elements of a predetermined granularity. For example, if the predetermined granularity is the granularity of one word and there are 10 words in the image to be analyzed, 10 image elements of the predetermined granularity can be determined.

[0038] The coordinate position of each image element of a predetermined granularity in the image to be analyzed is determined, and image layout information of the image to be analyzed is determined according to the coordinate positions of the image elements of the predetermined granularity in the image to be analyzed. The coordinate positions may be determined as image layout information, or the content represented by each image element of the predetermined granularity may be determined and associated with the corresponding coordinate positions to form image layout information. For example, if the image element of the predetermined granularity is a word, the word is associated with the coordinate position where the word is located and stored to obtain image layout information. Alternatively, if the image element of the predetermined granularity is a single tree, information such as the size and color of the tree is associated with the coordinate position of the tree and stored to obtain image layout information.

[0039] In this embodiment, the granularity of image elements is set in advance, the coordinate position of each image element is determined as image layout information, and the image to be analyzed is analyzed at the predetermined granularity, thereby improving the comprehensiveness of information extraction and improving the ability to understand images.

[0040] In this embodiment, the step of determining image elements of a predetermined granularity in the image to be analyzed includes a step of performing image recognition processing on the image to be analyzed according to the predetermined granularity, and obtaining image elements of the predetermined granularity in the image to be analyzed.

[0041] Specifically, the granularity of image elements is preset, and image recognition is performed on the image to be analyzed according to the preset granularity, recognizing all image elements of the predetermined granularity in the image to be analyzed. For example, if the predetermined granularity is one word, each word in the image to be analyzed can be recognized, and each word becomes an image element of the predetermined granularity. This embodiment does not particularly limit the predetermined granularity of the image elements, but it is necessary to ensure that image elements of the predetermined granularity have meaning; for example, one alphabet cannot be set as the predetermined granularity.

[0042] A layout recognition algorithm that can be used to recognize image elements of a predetermined granularity in the image to be analyzed can be preset. After all image elements of a predetermined granularity in the image to be analyzed are determined, semantic recognition may be performed on the image elements of the predetermined granularity to obtain the meanings of the image elements of the predetermined granularity. In this embodiment, the preset layout recognition algorithm is not particularly limited.

[0043] The beneficial effect of such a setting is that by recognizing the composition of fine-grained elements in the image to be analyzed based on pre-defined layout recognition techniques and obtaining multiple image elements of a predetermined granularity, it becomes easier to subsequently obtain image layout information for the image, improving the efficiency and accuracy of information extraction.

[0044] In this embodiment, the step of determining the coordinate positions of image elements of a predetermined granularity in the image to be analyzed includes the steps of performing image segmentation processing on the image to be analyzed according to the image elements of the predetermined granularity to obtain multiple image blocks, with one image block representing one image element of the predetermined granularity, and determining the coordinate positions of the image blocks in the image to be analyzed.

[0045] Specifically, after obtaining image elements of a predetermined granularity, an image segmentation process is performed on the image to be analyzed according to the image elements of the predetermined granularity, and a plurality of image blocks are divided from the image to be analyzed. One image block can represent one image element of the predetermined granularity, and for example, the image elements of the predetermined granularity can be divided in the form of a rectangular frame.

[0046] The coordinate position of each image block in the image to be analyzed is determined, and image layout information of the image to be analyzed is determined according to the coordinate position of the image block. The coordinate position may be determined as the image layout information, or both the image element in the image block and the corresponding coordinate position may be used as the image layout information of the image to be analyzed. For example, if the image element in the image block is a single word, the word can be associated with the coordinate position of the image block and stored as image layout information. Once the semantic information of the image element in the image block has been determined, the semantic information can also be associated with the corresponding coordinate position and stored as image layout information.

[0047] The beneficial effect of such a setting is that, for each image block, the contents in the image block and the coordinate position of the image block are determined as image layout information, thereby realizing correlation between the positions and contents of image elements of a predetermined granularity, avoiding confusion of contents between positions, and improving the accuracy of the image layout information.

[0048] In S202, text extraction processing of the text modality is performed on the image to be analyzed to obtain first content information corresponding to the text modality, and the content expressed by the visual modality in the image to be analyzed is converted into text written in natural language to obtain second content information corresponding to the visual modality.

[0049] For example, the image to be analyzed may include modality data, such as text modality and visual modality. After acquiring a multi-modality image to be analyzed, content information corresponding to various modalities can be obtained from the image to be analyzed. In the case of text modality, text extraction processing of the text modality is performed on the image to be analyzed, i.e., text information in the image to be analyzed is extracted, and first content information can be obtained. The first content information becomes the text information in the image to be analyzed. A character recognition algorithm can be preset to extract text; for example, an OCR (Optical Character Recognition) algorithm can be used. In this embodiment, the preset character recognition algorithm is not particularly limited.

[0050] In the case of a visual modality, the screen content of the image to be analyzed can be acquired as content expressed by the visual modality. For example, a background screen of text can be acquired, and all patterns other than text can be acquired from the image to be analyzed. The content expressed by the visual modality in the image to be analyzed can be converted into text written in natural language, and the text written in natural language can be used to explain the content expressed by the visual modality. The obtained text written in natural language is determined as second content information, i.e., the visual content of the image to be analyzed is converted into a caption (interpretation). An image description generation algorithm can be preset, and the second content information can be obtained according to the image description generation algorithm. In this embodiment, the preset image description generation algorithm is not particularly limited.

[0051] In this embodiment, based on technologies such as character recognition and graphic conversion recognition, content information corresponding to each modality in the image to be analyzed is extracted, thereby realizing comprehensive information extraction of the image to be analyzed, avoiding information loss, and improving the accuracy of image understanding, thereby improving the experience of human-machine interaction.

[0052] In S203, in response to question information from the user, response information corresponding to the question information is determined according to the image layout information and image content information, where the question information indicates a question posed by the user about the image to be analyzed, and the response information indicates a reply to the question indicated by the question information.

[0053] Illustratively, this step can refer to step S102 above, and will not be described again.

[0054] Figure 3 is a schematic diagram of the image to be analyzed. The image to be analyzed in Figure 3 has three lines of text on the left, trees on the right, and clouds in the background. That is, the image to be analyzed contains data of two modalities: text modality and visual modality. The text on the left is the text modality, and the trees on the right and the clouds in the background are the visual modality.

[0055] In an embodiment of the present disclosure, the image to be analyzed may include data from multiple modalities, such as text and visual modalities. By acquiring the image to be analyzed, image layout information and image content information of the image to be analyzed are determined. The image layout information can indicate the distribution of image elements of a predetermined granularity in the image to be analyzed, and the image content information can indicate the content expressed by the modality data in the image to be analyzed. By determining the image layout information and image content information, the image to be analyzed can be comprehensively understood and analyzed, improving the comprehensiveness of information extraction and comprehension ability. In response to a user's question, response information corresponding to the question is determined according to the layout information and content information. This solves the problem of the prior art being unable to understand complex multi-modality images, improves the accuracy of question answering, and enhances the human-machine interaction experience.

[0056] FIG. 4 is a flowchart of an image-based human-machine interaction method provided by an embodiment of the present disclosure.

[0057] In this embodiment, the step of determining response information corresponding to question information according to image layout information and image content information may be detailed as a step of determining the image type of the image to be analyzed according to image layout information and image content information, a step of determining semantic information of the question information and extracting target information corresponding to the semantic information from the image layout information and image content information, and a step of determining response information corresponding to the question information according to the target information and the image type of the image to be analyzed.

[0058] As shown in FIG. 4, the method includes S401 to S404.

[0059] In S401, the image to be analyzed is acquired, and image layout information and image content information of the image to be analyzed are determined. The image to be analyzed contains multiple types of modality data, and the image layout information indicates the distribution status of image elements of a specified granularity in the image to be analyzed, and the image content information indicates the content expressed by the modality data in the image to be analyzed.

[0060] Illustratively, this step can refer to the above step S101, and will not be described again.

[0061] In S402, the image type of the image to be analyzed is determined according to the image layout information and image content information.

[0062] For example, after obtaining the image layout information and the image content information, the image type of the analysis target image may be determined according to the image layout information and the image content information, such as a document screenshot, a table screenshot, a flowchart, an architecture diagram, or a data chart.

[0063] Rules for determining the image type can be set in advance. For example, if it is determined based on the image layout information that the image to be analyzed contains image elements of a predetermined granularity arranged in a multi-row, multi-column format, the image type of the image to be analyzed can be determined as a screenshot of a table. Also, for example, if it is determined based on the image layout information that the image to be analyzed contains long text information, and based on the image content information, it is determined that the meaning expressed by the text information is an article, the image type of the image to be analyzed can be determined as a screenshot of a document. In this embodiment, the rules for determining the image type are not particularly limited.

[0064] In this embodiment, the method further includes a step of determining a similarity between each of the image layout information, the first content information in the image to be analyzed, and the second content information, and if all of the similarities are equal to or greater than a predetermined similarity threshold, a step of determining an image type of the image to be analyzed according to the image layout information and the image content information.

[0065] Specifically, the image content information of the image to be analyzed may include first content information and second content information. For three types of information, i.e., layout information, first content information, and second content information, similarities between each pair of information are determined. That is, the similarity between the image layout information and the first content information, the similarity between the layout information and the second content information, and the similarity between the first content information and the second content information may be determined. These three types of information may be divided into two groups, each containing two types of information, and only two similarities may be determined. For example, the similarity between the image layout information and the first content information and the similarity between the image layout information and the second content information may be determined. In this embodiment, the method for calculating similarities is not particularly limited.

[0066] A similarity threshold is set in advance, and each obtained similarity is compared with the similarity threshold. If all of the obtained similarities are equal to or greater than the predetermined similarity threshold, the image layout information and the image content information match, and the image layout information and the image content information can be considered to be accurate information, and the image type of the image to be analyzed can be determined according to the image layout information and the image content information.

[0067] The beneficial effect of such a setting is that by determining the similarity, it is possible to determine whether the image layout information and image content information are accurate, which in turn improves the accuracy of subsequent classification decisions, thereby improving the accuracy of image classification decisions and responses to questions.

[0068] In this embodiment, the method further includes the steps of: responding when the similarity is smaller than a predetermined similarity threshold, determining standard information for adjusting the image layout information and / or image content information from among the image layout information, the first content information, and the second content information; determining the adjusted image layout information and image content information according to the standard information; and determining the image type of the image to be analyzed according to the adjusted image layout information and image content information.

[0069] Specifically, if there is one or more similarities that are smaller than a preset similarity threshold, it is determined that there is incorrect information in the image layout information, the first content information, and the second content information, and that the incorrect information needs to be adjusted.

[0070] First, standard information can be determined from the image layout information, the first content information, and the second content information. The standard information is accurate information and can be used to adjust the image layout information and / or the image content information, i.e., to adjust incorrect information. For example, if the standard information is image layout information and the first content information and the second content information are incorrect information, the image layout information can be used to adjust the first content information and the second content information.

[0071] Standard information may be preset from among image layout information and image content information; for example, second content information may be preset as standard information. A rule for determining standard information may be preset, such that standard information is determined according to the similarity between each pair of information, two types of information with the highest similarity are determined, and one type of information from these two types of information is selected as standard information. For example, if the image layout information is information A, the first content information is information B, and the second content information is information C, and the similarity between A and B is high and the similarity between A and C is low, either A or B may be selected as standard information, and information C may be adjusted according to the standard information. In this embodiment, the rule for determining standard information is not particularly limited.

[0072] After determining the standard information, adjusted image layout information and image content information are determined according to the standard information. If the standard information is image layout information, the image content information is adjusted; if the standard information is first content information, the image layout information and second content information are adjusted; if the standard information is second content information, the image layout information and first content information are adjusted. The information adjustment method can be preset, and when adjusting the information, the standard information can be replaced with incorrect information. For example, if the standard information is image layout information and the information to be adjusted is second content information, and the image layout information indicates that the image to be analyzed is a table-format image and the second content information indicates that the image to be analyzed is a flowchart, the flowchart in the second content information can be changed to a table image. After the adjustment is complete, accurate image layout information and image content information can be obtained.

[0073] The image type of the image to be analyzed is determined according to the adjusted image layout information and image content information. An image classification model is set up in advance, and the adjusted image layout information and image content information are input into the image classification model, and the picture type of the image to be analyzed is obtained through a comprehensive judgment of the model. For example, the image type may be a pure image, a table, a document, a flowchart, etc. The image classification model may be a pre-trained neural network model, and in this embodiment, the model structure of the image classification model is not particularly limited.

[0074] The beneficial effect of such a setting is that when the similarity between the information is small, the information can be corrected to improve the accuracy of the information, and then the image type can be determined to improve the accuracy of the image type.

[0075] In this embodiment, the step of determining the image type of the image to be analyzed in accordance with the image layout information and the image content information includes the steps of determining a positional arrangement rule in the image to be analyzed of image elements of a predetermined granularity in the image to be analyzed in accordance with the image layout information, the positional arrangement rule indicating an arrangement rule in the image to be analyzed of the coordinate positions of the image elements of the predetermined granularity; determining the image type of the image to be analyzed as a first image type in accordance with the positional arrangement rule; determining the image type of the image to be analyzed as a second image type in accordance with second content information in the image to be analyzed; and obtaining the image type of the image to be analyzed in response to a match between the first image type and the second image type.

[0076] Specifically, the image layout information includes coordinate positions of image elements of a predetermined granularity in the image to be analyzed, and a positional arrangement rule for the image elements of the predetermined granularity in the image to be analyzed can be determined according to the coordinate positions of the image elements of the predetermined granularity in the image to be analyzed. The positional arrangement rule can indicate the arrangement rule for the coordinate positions of the image elements of the predetermined granularity in the image to be analyzed; for example, if the image to be analyzed is a table image, the predetermined granularity is the size of one field in the table. According to the coordinate positions of the image elements of the predetermined granularity in the image to be analyzed, it can be determined that the image elements of the predetermined granularity will have a positional arrangement rule of multiple rows and multiple columns in the image to be analyzed.

[0077] The image type of the image to be analyzed can be determined according to a positional arrangement rule, and the image type determined according to the positional arrangement rule can be defined as the first image type; that is, the image type determined according to the image layout information can be defined as the first image type. For example, if image elements of a predetermined granularity exhibit a positional arrangement rule of multiple rows and multiple columns in the image to be analyzed, the first image type can be defined as a table type. The image type of the image to be analyzed can be determined according to second content information in the image to be analyzed, and the image type determined according to the second content information can be defined as the second image type. The second content information refers to information that interprets and explains the image content, and the second image type can be directly read from the second content information. For example, if the second content information clearly states that the image to be analyzed is a flowchart, the second image type can be defined as a flowchart type.

[0078] If the first image type and the second image type match, the image type of the image to be analyzed can be directly obtained. That is, either the first image type or the second image type is determined as the image type of the image to be analyzed. If the image layout information or image content information is adjusted, the first image type and the second image type can be determined respectively according to the adjusted image layout information and image content information, thereby obtaining the image type of the image to be analyzed.

[0079] The beneficial effect of such a setting is that by taking into consideration image layout information and image content information comprehensively, it is possible to avoid mistakes in determining the type based on a single piece of information, improve the accuracy of determining the type, improve the accuracy of responses to questions, and improve the experience of human-machine interaction.

[0080] In this embodiment, the method further includes a step of determining, in response to a mismatch between the first image type and the second image type, a target type from among the first image type and the second image type according to a preset priority as the image type of the image to be analyzed.

[0081] Specifically, if the first image type and the second image type do not match, the first image type or the second image type cannot be directly determined as the image type of the image to be analyzed. It is necessary to determine one image type from the first image type and the second image type as the target type, and then determine the target type as the image type of the image to be analyzed.

[0082] The priority between image layout information and image content information can be set in advance, the information with the highest priority can be determined according to the preset priority, and the image type determined according to the information with the highest priority can be determined as the target type. For example, if the second content information has the highest priority, the image type of the image to be analyzed can be obtained by determining the second image type as the target type.

[0083] The beneficial effect of such a setting is that when the two types of image types do not match, the type with the more reliable information is selected as the image type of the image to be analyzed, thereby improving the accuracy of determining the image type.

[0084] In S403, the semantic information of the question information is determined, and target information corresponding to the semantic information is extracted from the image layout information and image content information.

[0085] For example, question information from a user is acquired, and semantic recognition is performed on the question information to acquire semantic information of the question information. That is, the semantic information indicates the meaning of the question information. One semantic recognition model can be set in advance to determine the semantic information of the question information. In this embodiment, the preset semantic recognition model is not particularly limited.

[0086] When selecting target information from image layout information and image content information, corresponding information can be searched for in the image layout information and image content information according to semantic information, and used as the target information. That is, target information matching the semantic information is found from the image layout information and image content information. For example, semantic recognition can be performed on the image layout information and image content information to find information in the image layout information that corresponds to the semantic information, and information in the image content information that corresponds to the semantic information.

[0087] In S404, response information corresponding to the question information is determined in accordance with the target information and the image type of the image to be analyzed.

[0088] For example, after the image type and target information are obtained, a response may be made to question information in accordance with the target information and the image type of the image to be analyzed, and response information corresponding to the question information may be obtained. For example, the image type and target information may be determined as response information, or the image type and target information may be combined to obtain response information. The response information may be obtained based on a preset tool, and the preset tool may be, for example, a tool for generating graphs or a tool for creating a to-do list.

[0089] In this embodiment, the step of determining response information in accordance with the target information and the image type of the image to be analyzed includes the step of determining an information format corresponding to the image type of the image to be analyzed in accordance with a predetermined association relationship between the image type and the information format, and the step of generating response information corresponding to the question information based on the information format corresponding to the image type of the image to be analyzed in accordance with the target information.

[0090] Specifically, a relationship between an image type and an information format is set in advance, and the format of the response information may differ depending on the image type. In accordance with the image type of the analysis target image and the relationship set in advance between the image type and the information format, an information format corresponding to the image type of the analysis target image is determined as the target format.

[0091] After obtaining the target information, the format of the target information is adjusted according to the target format, and the adjusted information is used as the response information. That is, the content expressed by the response information is the content expressed by the target information, and the format of the response information is an information format corresponding to the image type of the image to be analyzed. For example, the target information can be adjusted to the format of a table, and the generated table can be displayed to the user as the response information.

[0092] A beneficial effect of such a setting is that response information can be obtained in different formats depending on the image type, which helps the user to understand the response information and improves the user experience.

[0093] FIG. 5 is a flowchart of human-machine interaction provided by this embodiment. In FIG. 5, a user inputs an image to be analyzed and question information. The image to be analyzed is a multi-modality image, and the question information can be used to request content extraction, picture answering, picture Q&A, chart answering, copywriting, etc., for the image to be analyzed. The image to be analyzed is observed and analyzed to obtain first content information, second content information, and image layout information. The obtained layout information, content information, and question information are input into a pre-defined large-scale model. The large-scale model performs Chain of Thought (CoT) inference, and response information is output using pre-defined tools. The Chain of Thought (CoT) inference refers to the process of decomposing a complex problem into smaller problems step by step and solving them sequentially by involving the large-scale model step by step. In this embodiment, the chain of thought inference process first corrects and adjusts the information, then determines the image type, then extracts target information, and finally obtains response information by referring to the question information specified by the user.

[0094] In an embodiment of the present disclosure, the image to be analyzed may include data from multiple modalities, such as text and visual modalities. By acquiring the image to be analyzed, image layout information and image content information of the image to be analyzed are determined. The image layout information can indicate the distribution of image elements of a predetermined granularity in the image to be analyzed, and the image content information can indicate the content expressed by the modality data in the image to be analyzed. By determining the image layout information and image content information, the image to be analyzed can be comprehensively understood and analyzed, improving the comprehensiveness of information extraction and comprehension ability. In response to a user's question, response information corresponding to the question is determined according to the image layout information and image content information. This solves the problem of the inability of conventional technologies to understand complex multi-modality images, improves the accuracy of question answering, and enhances the human-machine interaction experience.

[0095] 6 is a structural block diagram of an image-based human-machine interaction device provided by an embodiment of the present disclosure. For convenience of explanation, only parts related to the embodiment of the present disclosure are shown. Referring to FIG. 6, the image-based human-machine interaction device 600 includes an acquisition unit 601, a first determination unit 602, and a second determination unit 603.

[0096] The acquisition unit 601 is used to acquire an image to be analyzed, which includes at least two types of modality data; the first determination unit 602 is used to determine image layout information and image content information of the image to be analyzed, the image layout information indicating the distribution status of image elements of a predetermined granularity in the image to be analyzed, and the image content information indicating the content expressed by modality data in the image to be analyzed; The second determination unit 603 is used to respond to the acquired question information and determine response information corresponding to the question information according to the image layout information and the image content information, where the question information indicates a question posed about the image to be analyzed, and the response information indicates a reply answer corresponding to the question information.

[0097] FIG. 7 is a structural block diagram of an image-based human-machine interaction device provided by an embodiment of the present disclosure. As shown in FIG. 7, the image-based human-machine interaction device 700 includes an acquisition unit 701, a first determination unit 702, and a second determination unit 703, and the second determination unit 703 includes a type determination module 7031, a goal determination module 7032, and a response determination module 7033.

[0098] The type determination module 7031 is used to determine the image type of the analysis target image according to the image layout information and the image content information; The target determination module 7032 is used for determining semantic information of the question information, and extracting target information corresponding to the semantic information from the image layout information and the image content information; The response determination module 7033 is used to determine response information corresponding to the question information according to the target information and the image type of the image to be analyzed.

[0099] In one example, the first determining unit 702: an element determination module used to determine image elements of a predetermined granularity in the image to be analyzed, the image elements representing components of the image to be analyzed; a coordinate determination module used to determine the coordinate positions of the image elements of the predetermined granularity in the image to be analyzed; a layout determination module used for determining the image layout information according to the coordinate position.

[0100] In one example, the element determination module: The image recognition submodule is used to perform image recognition processing on the image to be analyzed according to the predetermined granularity, and to obtain image elements of the predetermined granularity in the image to be analyzed.

[0101] In one example, the coordinate determination module: an image segmentation sub-module used to perform image segmentation processing on the analysis target image according to the image elements of the predetermined granularity to obtain a plurality of image blocks, where one image block represents one image element of the predetermined granularity; and a position determination sub-module, used to determine the coordinate position of the image block in the image to be analyzed.

[0102] In one example, the at least two types of modality data include a text modality and a visual modality, and the first determining unit 702: a text extraction module used to perform text extraction processing of a text modality on the analysis target image to obtain first content information corresponding to the text modality; and a visual conversion module used to convert the content expressed by the visual modality in the image to be analyzed into text written in a natural language to obtain second content information corresponding to the visual modality.

[0103] In one example, the response determination module 7033: a format determination sub-module, which is used to determine an information format corresponding to the image type of the analysis target image according to a preset association relationship between the image type and the information format; and an information generating sub-module for generating response information corresponding to the question information according to the target information and based on an information format corresponding to the image type of the image to be analyzed.

[0104] In one example, the classification determination module: a similarity determination submodule used to determine the similarity between the image layout information, the first content information in the analysis target image, and the second content information; and a similarity judgment submodule, which is used to determine the image type of the image to be analyzed according to the image layout information and the image content information in response to the similarities being equal to or greater than a predetermined similarity threshold.

[0105] In one example, a standard determination sub-module, which is used to determine standard information for adjusting the image layout information and / or image content information from among the image layout information, the first content information, and the second content information in response to the similarity being smaller than a predetermined similarity threshold; It further includes an information adjustment submodule used to determine adjusted image layout information and image content information according to the standard information, and to determine the image type of the image to be analyzed according to the adjusted image layout information and image content information.

[0106] In one example, the type determination module 7031 a rule determination submodule used to determine a position arrangement rule in the analysis target image of image elements of a predetermined granularity in the analysis target image according to the image layout information, wherein the position arrangement rule indicates an arrangement rule in the analysis target image of coordinate positions of image elements of the predetermined granularity; a first determination sub-module used to determine the image type of the analysis target image as a first image type according to the position arrangement rule; a second determination sub-module used to determine the image type of the analysis target image as a second image type according to the second content information; and a type determination sub-module used to obtain the image type of the image to be analyzed in response to a match between the first image type and the second image type.

[0107] In one example, The image processing device further includes a target type determination unit that is used to determine a target type from among the first image type and the second image type as the image type of the image to be analyzed in accordance with a preset priority in response to a mismatch between the first image type and the second image type.

[0108] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device.

[0109] FIG. 8 is a structural block diagram of an electronic device provided by an embodiment of the present disclosure, where the electronic device may be a terminal device or a server. As shown in FIG. 8, the electronic device 800 includes at least one processor 802 and a memory 801 communicatively connected to the at least one processor 802, where the memory stores instructions executable by the at least one processor 802, and the instructions are executed by the at least one processor 802 to enable the at least one processor 802 to perform the image-based human-machine interaction method of the present disclosure.

[0110] The electronic device 800 further includes a receiver 803 and a transmitter 804. The receiver 803 is for receiving commands and data transmitted from other devices, and the transmitter 804 is for transmitting commands and data to external devices.

[0111] According to an embodiment of the present disclosure, the present disclosure further provides a non-transitory computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to cause a computer to perform the solution provided by any one of the above embodiments.

[0112] According to an embodiment of the present disclosure, the present disclosure further provides a computer program, the computer program being stored in a readable storage medium, and at least one processor of an electronic device being able to read the computer program from the readable storage medium, and when the at least one processor executes the computer program, the electronic device performs the solution provided by any one of the above embodiments.

[0113] 9 is a schematic block diagram of an exemplary electronic device 900 that can be used to implement embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components, their connections and relationships, and their functions illustrated herein are merely examples and are not intended to limit the description herein and / or the practice of the present disclosure as claimed.

[0114] 9, the device 900 includes a computing unit 901, which can perform various appropriate operations and processes based on a computer program stored in a read-only memory (ROM) 902 or loaded from a storage unit 908 into a random access memory (RAM) 903. The RAM 903 can further store various programs and data required for the operation of the device 900. The computing unit 901, the ROM 902, and the RAM 903 are connected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0115] Several components in device 900 are connected to I / O interface 905, including input units 906 such as a keyboard and a mouse, output units 907 such as various types of monitors and speakers, storage units 908 such as magnetic disks and optical disks, and communication units 909 such as a network card, modem, wireless communication transceiver, etc. The communication units 909 enable device 900 to exchange information / data with other devices via computer networks such as the Internet and / or various telecommunication networks.

[0116] The computing unit 901 may be any of a variety of general-purpose and / or special-purpose processing components having processing or computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units that execute machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs each of the methods and processes described above, such as the image-based human-machine interaction method. For example, in some embodiments, the image-based human-machine interaction method may be embodied as a computer software program tangibly contained in a machine-readable medium, such as the storage unit 908. In some embodiments, some or all of the computer program may be loaded and / or installed into the device 900 via the ROM 902 and / or the communication unit 909. When loaded into the RAM 903 and executed by the computing unit 901, the computer program may perform one or more steps of the image-based human-machine interaction method described above. Alternatively, in other embodiments, the computing unit 901 can be configured to perform the image-based human-machine interaction method using any other suitable means (eg, relying on firmware).

[0117] The various system and technical embodiments described herein may be implemented in a digital electronic circuit system, an integrated circuit system, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), an application specific standard product (ASSP), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may be embodied in one or more computer programs that may be executed and / or interpreted by a programmable system that includes at least one programmable processor, which may be a special purpose or general purpose programmable processor, that may receive data and instructions from a storage system, at least one input device, and at least one output device, and that may transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0118] Program code for implementing the methods of the present disclosure can be programmed using any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, so that when executed by the processor or controller, the program code performs the functions / operations shown in the flowcharts and / or block diagrams. The program code can be entirely executed on a machine, partially executed on a machine, partially executed on a machine as a separate software package and partially executed on a remote machine, or entirely executed on a remote machine or server.

[0119] In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain or store a program for use with, or in connection with, an instruction execution system, device, or apparatus. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium includes, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination thereof. More specific examples of machine-readable storage media include one or more wire-based electrical connections, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0120] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user, and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other types of devices can also provide interaction with a user; for example, the feedback provided to the user can be any form of sensing feedback (e.g., visual feedback, auditory feedback, or haptic feedback) and can receive input from the user in any form (including acoustic, speech, and tactile input).

[0121] The systems and techniques described herein may be implemented in a computing system including a back-end component (e.g., a data server), a computing system including a middleware component (e.g., an application server), a computing system including a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user interacts with embodiments of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0122] The computer system may include a client terminal and a server. The client terminal and the server are generally remote from each other and typically interact via a communication network. The relationship between the client terminal and the server is created by a computer program running on a corresponding computer and having a client terminal-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in a cloud computing service system and solves the drawbacks of traditional physical hosts and VPS servers (abbreviated as "Virtual Private Server" or "VPS"), such as difficult management and poor business scalability. The server may also be a server in a distributed system or a server combined with a blockchain.

[0123] It should be understood that steps can be rearranged, added, or deleted using the various forms of flow shown above. For example, the steps described in the present disclosure may be performed in parallel, sequentially, or in a different order, but this specification does not limit this, as long as the technical solution disclosed in the present disclosure can achieve the desired results.

[0124] The above-described embodiments of the present invention do not limit the scope of protection of the present disclosure. Those skilled in the art can make various modifications, combinations, subcombinations, and substitutions based on design requirements and other factors. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present disclosure should be included within the scope of protection of the present disclosure.

Claims

1. acquiring an image to be analyzed that includes at least two types of modality data; a step of determining image layout information and image content information of the image to be analyzed, wherein the image layout information indicates a distribution state of image elements of a predetermined granularity in the image to be analyzed, and the image content information indicates content expressed by modality data in the image to be analyzed; a step of determining response information corresponding to the question information according to the image layout information and the image content information in response to the acquired question information, wherein the question information indicates a question posed about the image to be analyzed, and the response information indicates a reply answer corresponding to the question information; The at least two types of modality data include a text modality and a visual modality, and the step of determining image layout information and image content information of the image to be analyzed includes: performing a text extraction process of a text modality on the image to be analyzed to obtain first content information corresponding to the text modality; and converting the content expressed by the visual modality in the image to be analyzed into text written in a natural language to obtain second content information corresponding to the visual modality; the image content information includes the first content information and the second content information, determining response information corresponding to the question information according to the image layout information and the image content information, determining an image type of the image to be analyzed according to the image layout information and the image content information; determining semantic information of the question information, and extracting target information corresponding to the semantic information from the image layout information and the image content information; determining the response information according to the target information and an image type of the image to be analyzed.

2. The step of determining image layout information and image content information of the analysis target image includes: determining image elements of a predetermined granularity in the image to be analyzed, the image elements representing components of the image to be analyzed; determining coordinate positions of the image elements of the predetermined granularity in the image to be analyzed; and determining the image layout information according to the coordinate positions.

3. The step of determining image elements of a predetermined granularity in the image to be analyzed includes:

3. The image-based human-machine interaction method according to claim 2, further comprising the step of performing image recognition processing on the image to be analyzed according to the predetermined granularity, and obtaining image elements of the predetermined granularity in the image to be analyzed.

4. The step of determining coordinate positions of the image elements of the predetermined granularity in the analysis target image includes: a step of performing an image segmentation process on the image to be analyzed according to the image elements of the predetermined granularity to obtain a plurality of image blocks, wherein one image block represents one image element of the predetermined granularity; and determining the coordinate positions of the image blocks in the image to be analyzed.

5. The step of determining the response information according to the target information and the image type of the analysis target image includes: determining an information format corresponding to the image type of the image to be analyzed according to a predetermined association relationship between the image type and the information format; and generating response information corresponding to the question information in accordance with the target information and based on an information format corresponding to the image type of the image to be analyzed.

6. The step of determining the image type of the analysis target image according to the image layout information and the image content information includes: determining a similarity between two of the image layout information, the first content information, and the second content information; and determining an image type of the image to be analyzed according to the image layout information and the image content information in response to determining that all of the similarities are equal to or greater than a predetermined similarity threshold.

7. determining standard information for adjusting at least one of the image layout information and the image content information from among the image layout information, the first content information, and the second content information in response to the similarity being smaller than a predetermined similarity threshold; determining adjusted image layout information and image content information according to the standard information; The image-based human-machine interaction method according to claim 6, further comprising: determining an image type of the analysis target image according to the adjusted image layout information and image content information.

8. The step of determining the image type of the analysis target image according to the image layout information and the image content information includes: a step of determining a position arrangement rule in the analysis target image of image elements of a predetermined granularity in the analysis target image according to the image layout information, the position arrangement rule indicating an arrangement rule in the analysis target image of coordinate positions of image elements of the predetermined granularity; determining an image type of the analysis target image as a first image type in accordance with the position arrangement rule; determining an image type of the analysis target image as a second image type in accordance with the second content information; and obtaining an image type of the image to be analyzed in response to a match between the first image type and the second image type.

9. 9. The image-based human-machine interaction method according to claim 8, further comprising: in response to a mismatch between the first image type and the second image type, determining a target type from among the first image type and the second image type according to a preset priority as the image type of the image to be analyzed.

10. an acquisition unit adapted to acquire an image to be analyzed, the image comprising at least two types of modality data; a first determination unit used to determine image layout information and image content information of the image to be analyzed, the image layout information indicating a distribution state of image elements of a predetermined granularity in the image to be analyzed, and the image content information indicating content expressed by modality data in the image to be analyzed; a second determination unit, which is used to respond to the acquired question information and determine response information corresponding to the question information according to the image layout information and the image content information, wherein the question information indicates a question posed about the analysis target image, and the response information indicates a reply answer corresponding to the question information; The at least two types of modality data include a text modality and a visual modality, and the first determining unit: a text extraction module used to perform text extraction processing of a text modality on the analysis target image to obtain first content information corresponding to the text modality; a visual conversion module used to convert content expressed by a visual modality in the analysis target image into text written in a natural language to obtain second content information corresponding to the visual modality; the image content information includes the first content information and the second content information, The second determination unit: a type determination module used to determine the image type of the analysis target image according to the image layout information and the image content information; a target determination module used for determining semantic information of the question information and extracting target information corresponding to the semantic information from the image layout information and the image content information; a response determination module used for determining the response information according to the target information and the image type of the image to be analyzed.

11. The first determination unit: an element determination module used to determine image elements of a predetermined granularity in the image to be analyzed, the image elements representing components of the image to be analyzed; a coordinate determination module used to determine the coordinate positions of the image elements of the predetermined granularity in the image to be analyzed; The image-based human-machine interaction device according to claim 10 , further comprising: a layout determination module used for determining the image layout information according to the coordinate positions.

12. The element determination module: The image-based human-machine interaction device according to claim 11, further comprising an element recognition submodule used to perform image recognition processing on the image to be analyzed according to the predetermined granularity, and obtain image elements of the predetermined granularity in the image to be analyzed.

13. The coordinate determination module: an image segmentation sub-module that performs image segmentation processing on the image to be analyzed according to the image elements of the predetermined granularity to obtain a plurality of image blocks, where one image block represents one image element of the predetermined granularity; and a position determination sub-module used to determine the coordinate position of the image block in the image to be analyzed.

14. The response determination module: a format determination sub-module, which is used to determine an information format corresponding to the image type of the analysis target image according to a preset association relationship between the image type and the information format; and an information generation submodule used for generating response information corresponding to the question information according to the target information and based on an information format corresponding to the image type of the image to be analyzed.

15. The type determination module: a similarity determination submodule used for determining similarities between the image layout information, the first content information, and the second content information; and a similarity judgment submodule used to determine an image type of the image to be analyzed according to the image layout information and the image content information in response to determining that all of the similarities are equal to or greater than a predetermined similarity threshold.

16. a standard determination submodule, which is used to determine standard information for adjusting at least one of the image layout information and the image content information from among the image layout information, the first content information, and the second content information, in response to the similarity being smaller than a predetermined similarity threshold; The image-based human-machine interaction device of claim 15, further comprising an information adjustment submodule used for determining adjusted image layout information and image content information according to the standard information, and determining an image type of the image to be analyzed according to the adjusted image layout information and image content information.

17. The type determination module: a rule determination submodule used to determine a position arrangement rule in the analysis target image of image elements of a predetermined granularity in the analysis target image according to the image layout information, wherein the position arrangement rule indicates an arrangement rule in the analysis target image of coordinate positions of image elements of the predetermined granularity; a first determination sub-module used to determine the image type of the analysis target image as a first image type according to the position arrangement rule; a second determination submodule used to determine the image type of the analysis target image as a second image type according to the second content information; and a type determination submodule used to obtain the image type of the image to be analyzed in response to a match between the first image type and the second image type.

18. 18. The image-based human-machine interaction device according to claim 17, further comprising a target type determination unit used for determining a target type from among the first image type and the second image type according to a preset priority as the image type of the image to be analyzed in response to a mismatch between the first image type and the second image type.

19. at least one processor; a memory communicatively coupled to the at least one processor, An electronic device, wherein the memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor is capable of performing the image-based human-machine interaction method described in any one of claims 1 to 4.

20. A non-transitory computer-readable storage medium having stored thereon computer instructions, the computer instructions being used to cause a computer to perform the image-based human-machine interaction method of any one of claims 1 to 4.

21. A computer program, which, when executed by a processor, causes the steps of the image-based human-machine interaction method according to any one of claims 1 to 4 to be implemented.

Images