Data processing method, electronic device, storage medium and program product

Abstract

The invention discloses a data processing method, electronic equipment, a storage medium and a program product, and relates to the field of large model technology and image data processing. The method comprises the steps that a target image corresponding to a preset object is acquired, and the target image comprises an image of the preset object and text information used for describing attribute features of the preset object; identifying the target image to obtain text information contained in the target image; and inputting the text information and the target image into a multi-modal large model, and performing information extraction on the target image by using the multi-modal large model to generate at least one piece of question and answer information. The technical problem that the image information mined from the image is inaccurate in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the field of large model technology and image data processing, and more specifically, to a data processing method, electronic device, storage medium and program product. Background Art

[0002] Common smart assistants, such as customer service robots on shopping platforms, are usually equipped with a database that automatically replies to user questions. A large amount of question-answer text can be stored in the database. When the smart assistant receives a question input by a user, it can automatically search for the corresponding answer from the database to facilitate the user to understand the currently viewed product. However, when generating question-answer text to build a database, it is usually limited to the pictures taken of the product, which may result in errors in the positioning of the subject, resulting in inaccurate image information mined from the picture. Summary of the invention

[0003] The embodiments of the present application provide a data processing method, an electronic device, a storage medium, and a program product to at least solve the technical problem of inaccurate image information mined from an image in the related art.

[0004] According to one aspect of an embodiment of the present application, a data processing method is provided, including: obtaining a target image corresponding to a preset object, wherein the target image includes an image of the preset object and text information for describing attribute characteristics of the preset object; recognizing the target image to obtain the text information contained in the target image; inputting the text information and the target image into a multimodal large model, extracting information from the target image using the multimodal large model, and generating at least one question and answer information, wherein the inquiry information in the at least one question and answer information is used to inquire about the attribute characteristics of the preset object, and the reply information in the at least one question and answer information is used to reply to the inquiry information.

[0005] According to one aspect of an embodiment of the present application, a data processing method is also provided, including: accepting current query information sent by a client, wherein the current query information is used to inquire about attribute characteristics of a target object; based on the current query information, obtaining target question and answer information matching the current query information from a first database, wherein the first database is used to store at least one question and answer information and a target cutting image generated by any of the above methods; generating current reply information corresponding to the current query information at least based on the target question and answer information; and sending the current reply information to the client.

[0006] According to one aspect of an embodiment of the present application, a data processing method is also provided, including: responding to an input instruction on an operation interface, displaying current query information on the operation interface, wherein the current query information is used to inquire about the attribute characteristics of a target object; responding to the query instruction on the operation interface, displaying current reply information corresponding to the current query information on the operation interface, wherein the current reply information is information generated based at least on target question and answer information corresponding to the current query information, the target question and answer information is question and answer information stored in a first database that matches the current query information, and the first database is used to store at least one question and answer information and a target cutting image generated by any of the above methods.

[0007] According to one aspect of an embodiment of the present application, a data processing method is also provided, including: obtaining current query information by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter includes the current query information, and the current query information is used to inquire about the attribute characteristics of the target object; based on the current query information, obtaining target question and answer information matching the current query information from a first database, wherein the first database is used to store at least one question and answer information and a target cutting image generated by any of the above methods; generating current reply information corresponding to the current query information at least based on the target question and answer information; outputting the current reply information by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the current reply information.

[0008] According to one aspect of an embodiment of the present application, a data processing device is provided, including: an image acquisition module, used to acquire a target image corresponding to a preset object, wherein the target image includes an image of the preset object and text information used to describe attribute characteristics of the preset object; an information recognition module, used to recognize the target image and obtain the text information contained in the target image; an information extraction module, used to input the text information and the target image into a multimodal large model, extract information from the target image using the multimodal large model, and generate at least one question and answer information, wherein the inquiry information in the at least one question and answer information is used to inquire about the attribute characteristics of the preset object, and the reply information in the at least one question and answer information is used to reply to the inquiry information.

[0009] According to one aspect of an embodiment of the present application, a data processing device is also provided, including: an information receiving module, used to receive current query information sent by a client, wherein the current query information is used to inquire about the attribute characteristics of a target object; an information acquisition module, used to obtain target question and answer information matching the current query information from a first database based on the current query information, wherein the first database is used to store at least one question and answer information and a target cutting image generated by any of the above methods; an information generation module, used to generate current reply information corresponding to the current query information based at least on the target question and answer information; and an information sending module, used to send the current reply information to the client.

[0010] According to one aspect of an embodiment of the present application, a data processing device is also provided, including: a first display module, used to respond to an input instruction acting on an operation interface, and display current query information on the operation interface, wherein the current query information is used to inquire about the attribute characteristics of a target object; a second display module, used to respond to the query instruction acting on the operation interface, and display current reply information corresponding to the current query information on the operation interface, wherein the current reply information is information generated based on at least target question and answer information corresponding to the current query information, and the target question and answer information is question and answer information stored in a first database that matches the current query information, and the first database is used to store at least one question and answer information and a target cutting image generated by any of the above methods.

[0011] According to one aspect of an embodiment of the present application, a data processing device is also provided, including: an information calling module, used to obtain current query information by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter includes the current query information, and the current query information is used to inquire about the attribute characteristics of the target object; an information matching module, used to obtain target question and answer information matching the current query information from a first database based on the current query information, wherein the first database is used to store at least one question and answer information and a target cutting image generated by any of the above methods; a reply generation module, used to generate current reply information corresponding to the current query information based at least on the target question and answer information; an information output module, used to output the current reply information by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the current reply information.

[0012] According to another aspect of the embodiments of the present application, an electronic device is further provided, including: a memory storing an executable program; and a processor for running the program, wherein the method of each embodiment of the present application is executed when the program is running.

[0013] According to another aspect of an embodiment of the present application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the methods in each embodiment of the present application.

[0014] According to another aspect of the embodiments of the present application, a computer program product is also provided, including a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present application is implemented.

[0015] According to another aspect of an embodiment of the present application, a computer program product is also provided, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present application is implemented.

[0016] According to another aspect of the embodiments of the present application, a computer program is also provided, and when the computer program is executed by a processor, the methods in the various embodiments of the present application are implemented.

[0017] In an embodiment of the present application, a target image corresponding to a preset object is obtained; the target image is recognized to obtain text information contained in the target image; the text information and the target image are input into a multimodal large model, and information is extracted from the target image using the multimodal large model to generate at least one question and answer information. Question and answer information is mined from the target image based on text information that can reflect the attribute characteristics of the preset object and for different attribute characteristics contained in the preset object. This ensures a degree of match between the generated question and answer information and the attribute characteristics of the preset object, thereby improving the accuracy of the question and answer information, thereby solving the technical problem of inaccurate image information mined from images in the related art.

[0018] It is easy to notice that the above general description and the following detailed description are only for the purpose of exemplifying and explaining the present application, and do not constitute a limitation of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 is a schematic diagram of a data processing scenario according to the present application;

[0021] Figure 2 is a flow chart of a data processing method according to the present application;

[0022] Figure 3is a flow chart of another data processing method according to the present application;

[0023] Figure 4 is a schematic diagram of a data processing process according to the present application;

[0024] Figure 5 is a flow chart of another data processing method according to the present application;

[0025] Figure 6 is a flow chart of another data processing method according to the present application;

[0026] Figure 7 is a structural block diagram of a data processing device according to the present application;

[0027] Figure 8 is a structural block diagram of another data processing device according to the present application;

[0028] Fig. 9 is a structural block diagram of another data processing device according to the present application;

[0029] Fig.10 is a structural block diagram of another data processing device according to the present application;

[0030] Fig.11 It is a structural block diagram of an electronic device according to the present application. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] The technical solution provided in this application is mainly implemented by large-scale model technology. The large model here refers to a deep learning model with large-scale model parameters, which can usually contain hundreds of millions, tens of billions, hundreds of billions, trillions or even more than 10 trillion model parameters. The large model can also be called a cornerstone model / foundation model (Foundation Model). The large model is pre-trained through large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as large-scale language model (Large Language Model, LLM), multi-modal pre-training model (multi-modal pre-training model), etc.

[0034] It should be noted that when the large model is actually applied, the pre-trained model can be fine-tuned through a small number of samples so that the large model can be applied to different tasks. For example, the large model can be widely used in natural language processing (NLP), computer vision, speech processing and other fields. Specifically, it can be applied to computer vision tasks such as visual question answering (VQA), image caption (IC), image generation, etc. It can also be widely used in natural language processing tasks such as text-based sentiment classification, text summary generation, machine translation, etc. Therefore, the main application scenarios of the large model include but are not limited to digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc. In an embodiment of the present application, taking the scenario where a user inquires about the purchase of a product when purchasing a product online, the information displayed by the product image is proposed through models such as multimodal processing models and multi-source question and answer models, and the user's questions are automatically replied according to the extraction results as an example for explanation,

[0035] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0036] Multimodal processing model: A natural language processing model that combines multiple data modalities (such as text, images, etc.) to understand and process the interaction of different types of data, achieving richer semantic representation and more accurate task execution.

[0037] According to an embodiment of the present application, a data processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0038] Considering the huge amount of model parameters of the large model and the limited computing resources of the mobile terminal, the above data processing method provided in the embodiment of the present application can be applied to Figure 1 The application scenarios shown are not limited to these. Figure 1 is a schematic diagram of a data processing scenario according to the present application. Figure 1 In the application scenario shown, the large model is deployed in the server 10, and the server 10 can be connected to one or more client devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client device 20 here may include but is not limited to: a smart phone, a tablet computer, a laptop computer, a PDA, a personal computer, a smart home device, a vehicle-mounted device, etc. The client device 20 can interact with the user through a graphical user interface to implement the call of the large model, thereby implementing the method provided in the embodiment of the present application.

[0039] In an embodiment of the present application, a system composed of a client device and a server can perform the following steps: the client device executes obtaining a target image corresponding to a preset object, receiving current query information, and outputting current reply information. The server performs optical character recognition on the target image, obtains text information contained in the target image, extracts information from the target image based on the text information, generates at least one question and answer information, stores at least one question and answer information and the target image in a first database, obtains target question and answer information from the first database according to the current query information, and generates current reply information at least according to the target question and answer information. It should be noted that, if the operating resources of the client device can meet the deployment and operating conditions of the large model, the embodiment of the present application can be performed in the client device.

[0040] Under the above operating environment, this application provides Figure 2 The data processing method shown. Figure 2 is a flow chart of a data processing method according to the present application. Figure 2 As shown, the method may include the following steps:

[0041] Step S202, obtaining a target image corresponding to a preset object.

[0042] The target image includes an image of a preset object and text information for describing attribute characteristics of the preset object.

[0043] The above-mentioned preset object may be an object to be displayed to the user, such as exhibits, commodities and other items. The above-mentioned target image may refer to an image constructed for the preset object, which is used to introduce the information contained in the preset object. The above-mentioned attribute characteristics may refer to the information contained in the preset object that needs to be introduced. The above-mentioned text information may refer to text information constructed for the attribute characteristics, which can be set by the user or generated directly by the processing system according to the attribute characteristics of the preset object, without specific limitation. For ease of understanding, taking the preset object as a cup sold on the shopping platform as an example, the above-mentioned target image may be the main picture of the cup taken by the merchant, as well as the product details page of the cup. The product details page may include text information introducing the attribute characteristics of the cup, which can facilitate users to understand the attribute characteristics of the cup, wherein the text information may include but is not limited to: the material, texture, quality, price, function and other characteristics of the cup.

[0044] In an optional solution of this embodiment, it is considered that when a user views a preset object, such as the above-mentioned commodity or exhibit, there are usually some questions related to the preset object that need to be inquired. For example, when a user views the above-mentioned cup, he may ask the customer service about the size, function, purchase discount policy, commodity delivery and after-sales service of the cup, or when viewing the cultural relics or paintings exhibited in the museum, he may ask the staff about the era, author, value and other issues of the cultural relics. The customer service or staff may not be able to answer all the user's questions accurately at any time and anywhere. Therefore, the corresponding reply content can be automatically queried according to the user input question through a database query method through an intelligent assistant such as a customer service robot on a shopping platform or a display operation table in a museum. Further, by performing statistical analysis on the questions input by the user in the customer service system, it can be known that most of the replies corresponding to the questions input by the user can be obtained through the above-mentioned target image containing the preset object. Therefore, the processing system can first extract and summarize the information of the target image containing the preset object to construct a database for answering the questions input by the user, and then the above-mentioned intelligent assistant automatically answers the questions input by the user through the constructed database to improve the question-answering effect of the intelligent assistant and ensure the user's experience. Based on this, when constructing a database, the processing system can first determine the above-mentioned preset object, and then obtain a target image matching the preset object. For example, if the above-mentioned preset object is a commodity, the above-mentioned target image can be obtained from the introduction page of the commodity, and contains the preset object and an image describing the attribute characteristics of the preset object, which can be directly intercepted and segmented from the introduction page. Considering that the attribute characteristics of the preset object may change, for example, when the preset object is a commodity, the price, preferential policy, sales volume and other characteristics of the commodity may change. If these characteristics are not updated in time in the constructed database, there may still be a situation where the reply is wrong. Therefore, in the case of a specified situation, for example, when an object that is not stored in the database appears, or the attribute characteristics of the object change, the processing system can first determine the object that is not stored in the data, or the object whose attribute characteristics have changed, as the above-mentioned preset object, and then obtain the target image corresponding to the preset object, so as to avoid the problem of incorrect reply to the user input due to untimely database update, or repeated database update, resulting in increased database construction costs.

[0045] Step S204: recognize the target image to obtain text information contained in the target image.

[0046] In an optional scheme of the present embodiment, in order to accurately reply to the user with the attribute characteristics of the preset object and avoid the situation where the constructed reply is inconsistent with the actual attribute characteristics of the preset object, for example, if the actual size of the preset object is A, and the display size of the preset object displayed in the target image is B, when the processing system determines the size of the preset object from the target image, it mistakenly determines the display size B as the actual size of the preset object. Based on this, after acquiring the target image, the processing system can first identify the text information contained in the target image to obtain the text information contained in the above target image, so that the processing system can accurately construct a database for replying to the questions input by the user based on the attribute characteristics clearly stated in the text information, thereby ensuring the accuracy of the intelligent assistant in replying to the questions input by the user.

[0047] Step S206: input the text information and the target image into the multimodal large model, extract information from the target image using the multimodal large model, and generate at least one question and answer information.

[0048] The inquiry information in at least one question-and-answer message is used to inquire about the attribute characteristics of the preset object, and the reply information in at least one question-and-answer message is used to reply to the inquiry information.

[0049] The above inquiry information may refer to a question input by a user, and the corresponding reply information may refer to an answer matching the question input by the user.

[0050] In an optional solution of the present embodiment, in order to quickly reply to the question input by the user, after the processing system recognizes the text information from the target image, it can directly extract information from the target image according to the text information, determine the image information matching the attribute features contained in the text information from the target image, so as to construct the inquiry information for inquiring about the attribute features of the preset object, and the reply information for replying to the inquiry information, and integrate the inquiry information and the reply information to generate the at least one question and answer information mentioned above. When the question input by the user is detected, the intelligent assistant can first recognize the question input by the user to obtain the question text, and then match the question text with the constructed inquiry information to obtain the matching degree between the question text and the inquiry information, and finally determine the reply information corresponding to the inquiry information with the largest matching degree as the answer to the question input by the user, which can improve the efficiency of replying to the question input by the user. Compared with first determining the keywords from the question text, and then matching the keywords with the attribute features in the text information one by one, and finally constructing the reply information according to the matching results, the efficiency of replying to the user is greatly improved, and the user experience is guaranteed. Moreover, the processing system generates the above-mentioned question and answer information based on the text information, which can ensure the relevance between the generated question and answer information and the text information, avoid generating question and answer information irrelevant to the preset object, and thus improve the accuracy, flexibility and comprehensiveness of the constructed question and answer information.

[0051] To facilitate understanding of the constructed question and answer information, for example, assuming that the text information indicates that the preset object contains structure A and introduces the function of structure A, the processing system can identify the corresponding structure A from the target image based on the text information, and determine image information related to structure A such as the position, texture, and functional principle of structure A on the preset object, and then construct question and answer information related to structure A based on this information. For example, assuming that the position of structure A on the preset object is position A, the inquiry information constructed for the position of structure A may include "Where is structure A", "What is the structure at position A in the figure", and "What are the functions of structure A", and the corresponding answers may be "Structure A is at position A on the preset object", "The structure at position A in the legend is structure A", and "What functions does structure A have".

[0052] In order to improve the accuracy of the generated question and answer information, when extracting information from the target image, the above-mentioned multimodal processing model can also be introduced, and the text information and the target image can be input into the multimodal processing model at the same time, and the multimodal processing model can generate different types of question and answer information for different attribute features of the preset object. For example, the processing system can first use the multimodal processing model to generate at least one reply information matching the attribute feature according to the attribute features in the text information and the performance results of these attribute features in the target image, such as the position, texture and other contents of the attribute features in the target image, and the user can understand the attribute feature through the reply information, and then can automatically generate the inquiry information matching the reply information according to the question text in the preset format, wherein the preset format of the question text can be input by the staff, or can be obtained from the Internet through big data statistics with the content input by multiple different users when asking questions, and the preset format of the question text is summarized according to these contents. After generating the inquiry information and reply information, the processing system can also display them to the staff in the preset operation interface, and the staff can adjust these information to ensure that the generated question and answer information is more in line with the user's daily life, thereby improving the user's experience when querying questions. It should be noted that the above-mentioned method of generating question and answer information is only an example and is not limited here.

[0053] In an optional scheme of the present embodiment, after generating the question and answer information, the processing system may store the question and answer information in a database, i.e., the first database mentioned above. When a question input by a user is detected, the intelligent assistant may quickly query the corresponding reply information from the first database. Considering that a simple text reply may cause misunderstanding or incomprehension to the user, while storing the question and answer information in the first database, the processing system may also store the target image in the first database. When the processing system queries the corresponding question and answer information according to the question input by the user, the reply information in the question and answer information may be displayed to the user. If the attribute features corresponding to the reply information can be reflected in the target image, the target image may also be displayed to the user at the same time to facilitate the user to understand the reply information, thereby improving the user experience.

[0054] In an embodiment of the present application, a target image corresponding to a preset object is obtained; the target image is recognized to obtain text information contained in the target image; the text information and the target image are input into a multimodal large model, and information is extracted from the target image using the multimodal large model to generate at least one question and answer information. Question and answer information is mined from the target image based on text information that can reflect the attribute characteristics of the preset object and for different attribute characteristics contained in the preset object. This ensures a degree of match between the generated question and answer information and the attribute characteristics of the preset object, thereby improving the accuracy of the question and answer information, thereby solving the technical problem of inaccurate image information mined from images in the related art.

[0055] In an embodiment of the present application, the above method also includes: cutting the target image based on the text information to obtain a target cut image; and storing at least one question and answer information and the target cut image in a first database.

[0056] In an optional solution of this embodiment, considering that the target image may contain attribute features of multiple preset objects, if the target image is directly displayed to the user at the same time as the reply information is displayed to the user, the user may not be able to determine the content related to the reply information from the target image, which affects the user experience. Therefore, when the question and answer information and the target image are actually stored in the first database, the target image can be cut according to the text information in the target image to obtain the above-mentioned target cut image, and then the question and answer information corresponding to the text information and the target cut image are stored in the first database in correspondence with each other, wherein different target cut images can correspond to different attribute features in the text information. For example, if the text information contains information A: "The handshake of the cup is arc-shaped", the target cut image cut according to the text information can be the area where the handshake of the cup is located in the target image, and the area where the information A is located. When the user queries for information related to the handshake of the cup, the processing system can directly display the reply information corresponding to the handshake of the cup and the target cut image to the user together, so as to facilitate the user to understand and view the handshake of the cup, thereby improving the user experience.

[0057] In an embodiment of the present application, the above method also includes: cutting the target image according to a preset size to obtain an initial cut image; identifying the target image to obtain text information contained in the target image, including: performing optical character recognition on the initial cut image to obtain text information; cutting the target image based on the text information to obtain a target cut image, including: cutting the target image based on the text information to obtain a target cut image.

[0058] The initial cut image may refer to the area where the text information is located in the target image.

[0059] In an optional solution of this embodiment, the text information in the assumed target image may include: information A, information B and information C. Considering that the processing system cannot directly segment the text information contained in the target image, if the text information is directly divided according to the font size, color and other features, a division error may occur, for example, part of the content in information A is divided into information B. Therefore, in order to facilitate the recognition of the text information in the target image, the multiple text information in the target image can be first divided into one area, that is, the above-mentioned initial cutting image. Based on this, before recognizing the target image, the processing system can first perform a preliminary recognition of the text contained in the target image, determine the area where the text information is located in the target image, and use the size of the area as the above-mentioned preset size, so as to use the preset size to cut the target image to obtain the above-mentioned initial cutting image, thereby improving the accuracy of the text information recognized from the target image, and reducing the influence of other information in the target image other than the text information on the recognition process, thereby improving the efficiency of recognizing the text information from the target image. It should be noted that the above-mentioned preset size can also be set by the staff, which is not limited here. The preset size is generally smaller than the image size of the target image.

[0060] After cutting out the initial cut image from the target image, in order to ensure the accuracy of the text information extracted from the target image, the processing system can further adopt optical character recognition (OCR) technology to identify the above text information from the initial cut image, and then use the identified text information to cut the target image again to obtain the above target cut image, thereby ensuring the accuracy of the cut target cut image, that is, the degree of matching between the target cut image and the text information.

[0061] In an embodiment of the present application, the above method further includes: converting the text information into a text image; and storing the text image, the initial cutting image and the target cutting image in a second database.

[0062] In an optional scheme of the present embodiment, considering that in real scenarios, the text information displayed in the target image may not be clearly displayed to the user due to loading failure, resolution abnormality, etc., resulting in the user being unable to intuitively determine the corresponding text information through the target image. Therefore, after identifying the text information, the processing system can also directly convert the identified text information into a corresponding text image, and then store the text image, the initial cut image containing the text information, and the target cut image matching the text information in the above-mentioned second database. When the user needs to view the text information contained in the target image, the processing system can directly display the corresponding text image, the initial cut image and the target cut image to the user, so as to utilize the text contained in the text image to assist in immediately viewing the text contained in the target image, thereby improving the user experience.

[0063] In an embodiment of the present application, after the target image is cut based on the text information, the method further includes: determining whether the target image is cut successfully to obtain a cutting result of the target image; and storing the cutting result of the target image in a first database.

[0064] In an optional solution of this embodiment, it is considered that when the target image is cut according to the text information, cutting errors may occur. For example, assuming that the text information describes "the cup has handshake A and handshake B, the color of handshake A is color A, and the color of handshake B is color B", the target cut image cut out from the target image by the corresponding processing system only includes the area where handshake A is located, that is, the attribute features contained in the target cut image do not match the attribute features contained in the text information. Therefore, after obtaining the target cut image, the processing system can also use the text information to verify the obtained target cut image to determine whether the target image is cut successfully, thereby obtaining the above-mentioned cutting. As a result, if the cutting result is unsuccessful, the processing system can re-cut the target image according to the text information to ensure the accuracy of the target cutting image finally obtained. In order to avoid showing the failed cutting image to the user, the processing system can also store the cutting result in the first database at the same time. If the cutting result shows that the cutting is successful, when displaying the reply information to the user, the processing system can display the target cutting image corresponding to the reply information to the user; if the cutting result shows that the cutting fails, the processing system can not display the target cutting image first, and use the target cutting image as training data to train the model used to cut the target cutting image from the target image, thereby improving the accuracy of the model.

[0065] In an embodiment of the present application, when it is determined that the target image cutting is successful, the above method also includes: updating the current number of cutting images corresponding to the preset object stored in the third database to obtain the target number of cutting images corresponding to the preset object, wherein the current number of cutting images is used to represent the number of historical cutting images corresponding to the preset object stored in the first database.

[0066] In an optional scheme of the present embodiment, if the cutting result is a successful cutting, the processing system can update the number of historical cutting images of the preset object stored in the first database, that is, the current number of cutting images, in the third database to obtain the target number of cutting images corresponding to the preset object, thereby avoiding frequent reading and writing of the first database when cutting the target cutting image, thereby improving the reading and writing efficiency of the first database.

[0067] In an embodiment of the present application, the above method also includes one of the following: when receiving target query information for inquiring about the attribute characteristics of a preset object, determining whether the target query information is preset query information, wherein the preset query information is used to characterize the query information received for the first time to inquire about the attribute characteristics of the preset object; when the target query information is not preset query information, determining whether at least one question and answer information is stored in the first database; when the target query information is preset query information, or at least one question and answer information is not stored in the first database, performing optical character recognition on the target image, extracting the target image based on text information, and storing at least one question and answer information and the target image in the first database.

[0068] In an optional solution of this embodiment, in order to avoid repeatedly constructing question and answer information, resulting in excessive waste of resources, and to avoid the absence of answers corresponding to the questions input by the user in the currently set question and answer information, resulting in an impact on the user experience, the processing system may determine whether the preset object is being asked by the user for the first time before constructing the question and answer information. That is, upon receiving the target inquiry information input by the user or staff for inquiring about the attribute characteristics of the preset object, it may determine whether the attribute characteristics of the preset object are being asked by the user for the first time. If so, it means that the preset object is an item that appears for the first time. At this time, the processing system may perform the aforementioned optical character recognition, text information extraction, target image cutting, and question and answer information construction on the target image corresponding to the preset object. And the operation of storing the question and answer information and the target image in the first database; if not, it means that the preset object may have been subject to knowledge mining. At this time, the target query information can be further matched with the query information stored in the first database to determine whether there is reply information matching the target question and answer information in the currently stored question and answer information. If so, it means that the processing system has constructed the corresponding question and answer information for the target query information, and the reply information contained in the question and answer information can be directly output. If not, it means that the question and answer information currently constructed for the attribute features of the preset object is incomplete. At this time, the processing system can re-execute the above-mentioned process of constructing and storing question and answer information for the target query information to improve the completeness of the constructed question and answer information.

[0069] In an embodiment of the present application, the above method also includes one of the following: when receiving change information for changing an original image of a preset object to a target image, determining whether the original image is stored in the first database; when the original image is stored in the first database, performing optical character recognition on the target image, extracting the target image based on text information, and storing at least one question and answer information and the target image in the first database.

[0070] In an optional scheme of the present embodiment, taking into account that the attribute characteristics of the preset object may change at any time, for example, in a shopping platform, the price, style, remaining quantity and other characteristics of the goods may change. If the constructed question and answer information is not adjusted in time for these changes, wrong reply information may be displayed to the user, resulting in a situation where the user experience is affected. Therefore, when receiving a change in the attribute characteristics of the preset object, that is, when receiving the change information for changing the original image of the preset object to the target image, the processing system can first determine whether the original image of the preset object is stored in the first database. If so, it means that the processing system has constructed corresponding question and answer information for the text information contained in the original image, and the corresponding question and answer information needs to be adjusted. At this time, the processing system can re-perform the aforementioned optical character recognition, text information extraction, target image cutting, question and answer information construction, and storage of the question and answer information and the target image in the first database for the changed target image, so as to ensure the timeliness of the question and answer information. In order to improve efficiency, the newly constructed question and answer information can be based on the question and answer information corresponding to the original image. For example, the new attribute features identified from the target image can be directly used to replace the old attribute features in the question and answer information to improve the efficiency of constructing the question and answer information. If not, the processing system can directly execute the above process for the new target image to generate new question and answer information to avoid the situation where the reply information matching the target query information cannot be displayed to the user, thereby improving the user experience.

[0071] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0072] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0073] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0074] According to an embodiment of the present application, another data processing method is also provided: Figure 3 is a flow chart of another data processing method according to the present application, such as Figure 3 As shown, the method may include the following steps:

[0075] Step S302: accept the current query information sent by the client.

[0076] The current query information is used to query the attribute characteristics of the target object.

[0077] The target object may be an object displayed to the user, such as exhibits, commodities, etc., and the preset objects mentioned above include the target object. The current inquiry information may refer to questions related to the target object input by the user, and may be used to inquire about the attribute characteristics of the target object.

[0078] In an optional scheme of the present embodiment, considering that when the user is viewing the target object, he may ask the staff some questions related to the target object, and the staff may not be able to answer the user anytime and anywhere, therefore, through intelligent assistants such as customer service robots on shopping platforms, display workstations in museums, etc., the corresponding reply content can be automatically queried according to the questions input by the user through database query. Based on this, when the user has relevant questions about the target object, the corresponding inquiry information, that is, the above-mentioned current inquiry information, can be input to the client, such as the customer service robot on the shopping platform, the display workstation in the museum and other equipment. The processing system can receive the current inquiry information sent from the client, and display the corresponding reply information to the user based on the current inquiry information, so as to answer the questions asked by the user.

[0079] Step S304: based on the current query information, obtaining target question and answer information matching the current query information from the first database.

[0080] The first database is used to store at least one question and answer information and a target cutting image generated by any one of the above methods.

[0081] The above-mentioned first database may refer to a database constructed by performing optical character recognition, text information extraction, target image segmentation, question and answer information construction, and corresponding storage of the question and answer information and the target image according to the aforementioned process.

[0082] In an optional solution of this embodiment, after receiving the current query information, the processing system may match the current query information with the question and answer information stored in the first database to determine the target question and answer information that matches the current query information. For example, the processing system may first perform keyword extraction on the current query information to preliminarily determine the attribute characteristics of the target object that the user wants to inquire about, and then filter out multiple question and answer information related to the attribute characteristics from the first database, and determine the query information contained in the multiple question and answer information, and then match the current query information with the multiple query information to determine the correlation between the current query information and the different query information stored in the first database, and finally determine the question and answer information corresponding to the query information with the highest correlation as the above-mentioned target question and answer information, thereby ensuring that the determined target question and answer information matches the current query content of the user and improving the accuracy of the target question and answer information. It should be noted that the correlation mentioned here, and the determination of the target question and answer information based on the correlation, are only exemplary displays and are not limited here.

[0083] Step S306: Generate current reply information corresponding to the current inquiry information based at least on the target question and answer information.

[0084] In an optional scheme of the present embodiment, after determining the target question and answer information, the processing system can determine the reply information contained in the target question and answer information, and determine the reply information as the answer to the content currently asked by the user, so as to generate the current reply information corresponding to the above-mentioned current inquiry information. Considering that different users may have different expression habits, it may happen that the meaning contained in the determined target question and answer information deviates from the meaning actually contained in the current inquiry information, resulting in the determined current reply information not being the content that the user wants to know. Therefore, in addition to directly determining the reply information contained in the target question and answer information as the above-mentioned current reply information, the processing system can also use the remaining multiple question information related to the current inquiry information as alternative question and answer information based on the above-mentioned determined correlation, and generate the above-mentioned current reply information together with the above-mentioned target question and answer information, thereby providing users with multiple reference information to understand the attribute characteristics of the target object.

[0085] Step S308: Send the current reply information to the client.

[0086] In an optional scheme of this embodiment, after generating the current reply information, the processing system can send the current reply information to the client so that the user can view the attribute characteristics of the target object. At the same time, the processing system can continue to receive query requests sent from the client and provide relevant content to the user.

[0087] In an embodiment of the present application, current reply information corresponding to the current query information is generated at least based on the target question and answer information, including: based on the current query information, obtaining multiple description information corresponding to the current query information from multiple data sources, wherein different description information is used to characterize information from different data sources that describes the attribute characteristics of the target object; merging the target question and answer information and the multiple description information to obtain a merged result; inputting the merged result into a multi-source question and answer model, and obtaining the current reply information output by the multi-source question and answer model.

[0088] The above-mentioned data source may refer to a data source containing information related to the attribute characteristics of the target object, and may include but is not limited to: information related to the attribute characteristics of the target object searched through a search engine, information related to the attribute characteristics of the target object provided by other users, and configuration, maintenance, and other information related to the attribute characteristics of the target object provided by the displayer of the target object, such as a merchant, museum, etc.

[0089] In an optional scheme of the present embodiment, considering that the text information directly extracted from the target image contains limited content, the reply information directly generated based on the text information may not be able to fully answer the query information input by the user. Therefore, in order to ensure the rationality and comprehensiveness of the current reply information generated, the processing system can first obtain multiple descriptive information for describing the attribute characteristics of the target object from the above-mentioned multiple data sources based on the current query information. For example, if the current query information is information input by the user to determine the function of attribute characteristic A, and there are currently data sources 1, 2 and 3, the processing system can obtain information related to the function of attribute characteristic A from these three data sources to obtain information set 1 corresponding to data source 1, information set 2 corresponding to data source 2, and information set 3 corresponding to data source 3. The information contained in these three information sets can be regarded as the above-mentioned descriptive information. After determining the descriptive information, the processing system can merge the descriptive information to remove abnormal information such as repeated content, low relevance to the attribute characteristics of the target object currently viewed by the user, and unclear meaning that makes it incomprehensible to the user, thereby obtaining the above-mentioned merged result. Finally, the merged result is input into the pre-trained multi-source question-answering model to obtain the current reply information that can help the user understand the attribute characteristics of the target object.

[0090] In an embodiment of the present application, when the target question and answer information is not obtained from the first database, the method also includes: obtaining a preset image corresponding to the target object, wherein the preset image includes an image of the target object and text information used to describe the attribute characteristics of the target object; identifying the current inquiry information and the preset image to determine the attribute characteristics of the target object; obtaining multiple descriptive information matching the attribute characteristics of the target object from multiple data sources; and generating current reply information based on the multiple descriptive information.

[0091] In an optional solution of this embodiment, if the target question and answer information matching the current query information is not obtained in the first database, it means that the question and answer information related to the attribute characteristics of the target object currently stored in the first database may not be updated in time. In order not to display the wrong reply information to the user, the processing system can first obtain a preset image containing the target object and text information describing the attribute characteristics of the target object through the method of image and text positioning, such as the product image displayed to the user in the product details page, and then recognize the current query information and the preset image to determine the attribute characteristics of the target object, such as by optical character recognition, determine the text information contained in the preset image, and determine the attribute characteristics of the target object through the text information and the current query information. After determining the attribute characteristics, the processing system can obtain multiple description information matching the determined attribute characteristics from multiple data sources, and regenerate the current reply information in the above-mentioned manner based on the obtained description information to ensure the rationality and accuracy of the generated current reply information.

[0092] In order to facilitate understanding of the current inquiry information input by the above-mentioned reply user and generate the content of the question and answer information matching the attribute characteristics of the target object, Figure 4 is a schematic diagram of a data processing process according to the present application, such as Figure 4 As shown, in general, the above data processing process can be divided into two parts: Figure 4The 401 in the figure represents the first part, which may refer to the real-time question and answer part, i.e., the intelligent assistant queries and replies to the user in real time according to the inquiry information input by the user, and it is carried out in real time. The 402 represents the part of offline database construction, i.e., the intelligent assistant determines whether it is necessary to generate corresponding question and answer information for the attribute characteristics of the preset object to construct the first database, the second database and the third database, which may be carried out offline. In the real-time question and answer part 401, the processing system first executes 4011, that is, receives the current query information input by the user, and then executes 4012 to determine whether the current query information is the preset query information. If so, it can further execute 4013 to determine whether target question and answer information matching the current query information is stored in the first database. If so, it means that knowledge mining has been performed on the target object currently inquired by the user, and correspondingly, 4014 can be executed to directly output the current reply information based on the target question and answer information; if the current query information is not the preset query information, or there is no question and answer information matching the current query information stored in the first database, the processing system can execute 4015 to first determine the attribute characteristics of the target object through image and text positioning according to the text information contained in the preset image corresponding to the target object, and then execute 4016 to generate the current reply information based on the relevant description of the attribute characteristics in the text information, and at the same time start to update the corresponding database for the attribute characteristics of the target object.In the part 402 of offline database construction, the processing system may first execute 4021 to obtain a target image of a preset object, wherein the preset object may include a target object. After obtaining the target image, the processing system may execute 4022 to determine whether the target image triggers a condition for updating the database, such as when the current query information mentioned above is not the preset query information, or when there is no question-and-answer information matching the current query information stored in the first database, or when the target image corresponding to the preset object changes. If there is no trigger, it means that image information mining is not required at present, and 4023 may be executed to not mine image information for the target image. If there is a trigger, the processing system may first execute 4024 to cut the target image according to a preset size to obtain an initial cut image. In the initial cut image, the processing system may execute 4025 to determine text information for describing the attribute characteristics of the preset object by means of optical character recognition, and then execute 4026 to further cut the target image according to the text information, for example, by using an EAS algorithm (Effective Adaptive Scaling). Sampling, an image cutting algorithm), cuts out areas corresponding to different attribute features that match the text information from the target image to obtain a target cut image. At the same time, 4027 can be executed to use a multimodal processing model to extract information from the target image based on the identified text information to generate corresponding question and answer information for different attribute features. Finally, 4028 is executed to store the generated question and answer information and the target cut image obtained by cutting into a first database. In order to manage these multiple images, the processing system can set up a second database to store the text information, the initial cut image and the target cut image generated in the above process into the second database. When storing the text information in the second database, 4029 can be executed first to convert the text information into a corresponding text image, and then 4030 can be executed to store the text image, as well as the corresponding initial cut image and the target cut image into the second database. At the same time, in order to facilitate the determination of the update status of the target image, a third database can also be set up to store the number of historical cutting images corresponding to the preset object. The processing system can execute 4031 to update the third database according to the number of cutting images generated in the above process. The processing system can filter the received information related to the target image according to the change in the number, for example, filtering out redundant messages such as repeated establishment of question and answer information and repeated updating of the first database, while retaining the information on changes in the target image, so that the processing system can timely update the attribute features that have changed in the preset object.

[0093] According to an embodiment of the present application, another data processing method is also provided: Figure 5 is a flow chart of another data processing method according to the present application, such as Figure 5 As shown, the steps of the method may include:

[0094] Step S502, in response to an input instruction on the operation interface, display current query information on the operation interface.

[0095] The current query information is used to query the attribute characteristics of the target object.

[0096] The input instruction may refer to an instruction for the user to input the current inquiry information on the operation interface. The target object may refer to an object displayed to the user, such as exhibits, commodities and other items. The current inquiry information may refer to a question related to the target object input by the user, which may be used to inquire about the attribute characteristics of the target object.

[0097] In an optional scheme of this embodiment, when the user enters the above-mentioned input instruction on the operation interface, the processing system can first display the above-mentioned current inquiry information to the user in the operation interface to facilitate the user's viewing and avoid user input errors, which may cause the current reply information queried to be not the information the user wants.

[0098] Step S504, in response to the inquiry instruction on the operation interface, current reply information corresponding to the current inquiry information is displayed on the operation interface.

[0099] Among them, the current reply information is information generated based at least on the target question and answer information corresponding to the current inquiry information, the target question and answer information is question and answer information stored in the first database that matches the current inquiry information, and the first database is used to store at least one question and answer information and target cutting image generated by any of the above methods.

[0100] The query instruction may refer to an instruction input by a user for querying the attribute characteristics of a target object. The first database may refer to a database constructed by performing optical character recognition, text information extraction, target image segmentation, question and answer information construction, and corresponding storage of the question and answer information and the target image according to the aforementioned process.

[0101] In an optional scheme of the present embodiment, when the user determines that the current inquiry information input is accurate and enters the above-mentioned inquiry instruction on the operation interface, the processing system can first obtain the target question and answer information matching the current inquiry information from the above-mentioned first database, and then generate the above-mentioned current reply information at least based on the determined target question and answer information. Finally, the processing system can display the current reply information to the user on the above-mentioned operation interface for the convenience of user viewing.

[0102] According to an embodiment of the present application, another data processing method is also provided: Figure 6is a flow chart of another data processing method according to the present application, such as Figure 6 As shown, the steps of the method may include:

[0103] Step S602: Acquire current query information by calling the first interface.

[0104] The first interface includes a first parameter, a parameter value of the first parameter includes current query information, and the current query information is used to query the attribute characteristics of the target object.

[0105] The first interface may refer to an interface for obtaining a parameter value of the first parameter, i.e., the current query information. The target object may be an object displayed to the user, such as exhibits, commodities, etc., and the preset object mentioned above includes the target object. The current query information may refer to a question related to the target object input by the user, which may be used to inquire about the attribute characteristics of the target object.

[0106] In an optional scheme of this embodiment, when the user has questions related to the target object, he can input corresponding inquiry information, that is, the above-mentioned current inquiry information, to the client, such as the customer service robot on the shopping platform, the display console in the museum and other equipment. The processing system can obtain the current inquiry information sent from the client through the above-mentioned first interface, and display corresponding reply information to the user based on the current inquiry information to answer the questions asked by the user.

[0107] Step S604: based on the current query information, obtaining target question and answer information matching the current query information from the first database.

[0108] The first database is used to store at least one question and answer information and a target cutting image generated by any one of the above methods.

[0109] The above-mentioned first database may refer to a database constructed by performing optical character recognition, text information extraction, target image segmentation, question and answer information construction, and corresponding storage of the question and answer information and the target image according to the aforementioned process.

[0110] In an optional solution of this embodiment, after receiving the current query information, the processing system may match the current query information with the question and answer information stored in the first database to determine the target question and answer information that matches the current query information. For example, the processing system may first perform keyword extraction on the current query information to preliminarily determine the attribute characteristics of the target object that the user wants to inquire about, and then filter out multiple question and answer information related to the attribute characteristics from the first database, and determine the query information contained in the multiple question and answer information, and then match the current query information with the multiple query information to determine the correlation between the current query information and the different query information stored in the first database, and finally determine the question and answer information corresponding to the query information with the highest correlation as the above-mentioned target question and answer information, thereby ensuring that the determined target question and answer information matches the current query content of the user and improving the accuracy of the target question and answer information. It should be noted that the correlation mentioned here, and the determination of the target question and answer information based on the correlation, are only exemplary displays and are not limited here.

[0111] Step S606: Generate current reply information corresponding to the current inquiry information based at least on the target question and answer information.

[0112] In an optional scheme of the present embodiment, after determining the target question and answer information, the processing system can determine the reply information contained in the target question and answer information, and determine the reply information as the answer to the content currently asked by the user, so as to generate the current reply information corresponding to the above-mentioned current inquiry information. Considering that different users may have different expression habits, it may happen that the meaning contained in the determined target question and answer information deviates from the meaning actually contained in the current inquiry information, resulting in the determined current reply information not being the content that the user wants to know. Therefore, in addition to directly determining the reply information contained in the target question and answer information as the above-mentioned current reply information, the processing system can also use the remaining multiple question information related to the current inquiry information as alternative question and answer information based on the above-mentioned determined correlation, and generate the above-mentioned current reply information together with the above-mentioned target question and answer information, thereby providing users with multiple reference information to understand the attribute characteristics of the target object.

[0113] Step S608: output the current reply information by calling the second interface.

[0114] The second interface includes a second parameter, and a parameter value of the second parameter includes current reply information.

[0115] The second interface mentioned above may refer to an interface for obtaining a parameter value of a second parameter, that is, the current reply information.

[0116] In an optional solution of this embodiment, after generating the current reply information, the processing system may output the current reply information in a preset operation interface by calling the above-mentioned second interface to facilitate user viewing.

[0117] According to an embodiment of the present application, a data processing device for implementing the above data processing method is also provided. Figure 7 is a structural block diagram of a data processing device according to the present application, such as Figure 7 As shown, the device includes: an image acquisition module 702, an information recognition module 704 and an information extraction module 706.

[0118] Among them, the image acquisition module 702 is used to obtain the target image corresponding to the preset object, wherein the target image includes the image of the preset object and text information used to describe the attribute characteristics of the preset object; the information recognition module 704 is used to recognize the target image and obtain the text information contained in the target image; the information extraction module 706 is used to input the text information and the target image into the multimodal large model, use the multimodal large model to extract information from the target image, and generate at least one question and answer information, wherein the inquiry information in at least one question and answer information is used to inquire about the attribute characteristics of the preset object, and the reply information in at least one question and answer information is used to reply to the inquiry information; the data storage module 708 is used to store at least one question and answer information and the target image in the first database.

[0119] In an embodiment of the present application, the above-mentioned device also includes: an image cutting module, which is used to cut the target image based on text information to obtain a target cut image; an information storage module, which is used to store at least one question and answer information and the target cut image into a first database.

[0120] In an embodiment of the present application, the above-mentioned device also includes: an initial cutting module, which is used to cut the target image according to a preset size to obtain an initial cut image; the information recognition module 704 is also used to: perform optical character recognition on the initial cut image to obtain text information; the image cutting unit is also used to: cut the target image based on the text information to obtain a target cut image.

[0121] In an embodiment of the present application, the above-mentioned device also includes: a text conversion module, which is used to convert text information into a text image; and a first storage module, which is used to store the text image, the initial cutting image and the target cutting image in a second database.

[0122] In an embodiment of the present application, the above-mentioned device also includes: a result determination unit, used to determine whether the target image is cut successfully and obtain the cutting result of the target image; a second storage module, used to store the cutting result of the target image in the first database.

[0123] In an embodiment of the present application, the above-mentioned device also includes: a skilled update module, which is used to update the current number of cutting images corresponding to the preset object stored in the third database to obtain the target number of cutting images corresponding to the preset object, wherein the current number of cutting images is used to represent the number of historical cutting images corresponding to the preset object stored in the first database.

[0124] In an embodiment of the present application, the above-mentioned device also includes one of the following: a first determination module, which is used to determine whether the target query information is preset query information when receiving target query information for inquiring about the attribute characteristics of a preset object, wherein the preset query information is used to characterize the query information for inquiring about the attribute characteristics of the preset object received for the first time; a second determination module, which is used to determine whether at least one question and answer information is stored in the first database when the target query information is not the preset query information; a first generation and storage module, which is used to perform optical character recognition on the target image, extract the target image based on text information, and store at least one question and answer information and the target image in the first database when the target query information is the preset query information or at least one question and answer information is not stored in the first database.

[0125] In an embodiment of the present application, the above-mentioned device also includes one of the following: a third determination module, which is used to determine whether the original image is stored in the first database when receiving change information for changing the original image of the preset object to the target image; a second generation and storage module, which is used to perform optical character recognition on the target image, extract the target image based on text information, and store at least one question and answer information and the target image in the first database when the original image is stored in the first database.

[0126] It should be noted that the image acquisition module 702, information identification module 704, information extraction module 706 and data storage module 708 correspond to steps S202 to S208 in the above embodiment, and the four modules and corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also be run in the server 10 provided in the above embodiment as part of the device.

[0127] According to an embodiment of the present application, a data processing device for implementing the above data processing method is also provided. Figure 8 is a structural block diagram of another data processing device according to the present application, such as Figure 8As shown, the device includes: an information receiving module 802 , an information acquiring module 804 , an information generating module 806 and an information sending module 808 .

[0128] Among them, the information receiving module 802 is used to receive the current query information sent by the client, wherein the current query information is used to inquire about the attribute characteristics of the target object; the information acquisition module 804 is used to obtain target question and answer information matching the current query information from the first database based on the current query information, wherein the first database is used to store at least one question and answer information and a target cutting image generated by any of the above methods; the information generation module 806 is used to generate current reply information corresponding to the current query information based at least on the target question and answer information; the information sending module 808 is used to send the current reply information to the client.

[0129] In an embodiment of the present application, the information generation module 806 includes: a first acquisition unit, which is used to acquire multiple descriptive information corresponding to the current query information from multiple data sources based on the current query information, wherein different descriptive information is used to characterize information from different data sources that describes the attribute characteristics of the target object; an information merging unit, which is used to merge the target question and answer information and the multiple descriptive information to obtain a merged result; and a second acquisition unit, which is used to input the merged result into a multi-source question and answer model and obtain the current reply information output by the multi-source question and answer model.

[0130] In an embodiment of the present application, the above-mentioned device also includes: a preset image acquisition module, which is used to acquire a preset image corresponding to the target object, wherein the preset image includes an image of the target object and text information used to describe the attribute characteristics of the target object; a preset image recognition module, which is used to recognize the current query information and the preset image to determine the attribute characteristics of the target object; a description information acquisition module, which is used to acquire multiple description information matching the attribute characteristics of the target object from multiple data sources; and a reply information generation module, which is used to generate current reply information based on multiple description information.

[0131] It should be noted that the above-mentioned information receiving module 802, information acquisition module 804, information generation module 806 and information sending module 808 correspond to steps S302 to S308 in the above-mentioned embodiment, and the four modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned embodiment. It should be noted that the above-mentioned modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above-mentioned modules can also be run in the server 10 provided in the above-mentioned embodiment as part of the device.

[0132] According to an embodiment of the present application, a data processing device for implementing the above data processing method is also provided. Fig. 9 is a structural block diagram of another data processing device according to the present application, such as Fig. 9 As shown, the device includes: a first display module 902 and a second display module 904 .

[0133] Among them, the first display module 902 is used to respond to the input instructions acting on the operation interface, and display the current query information on the operation interface, wherein the current query information is used to inquire about the attribute characteristics of the target object; the second display module 904 is used to respond to the query instructions acting on the operation interface, and display the current reply information corresponding to the current query information on the operation interface, wherein the current reply information is information generated based on at least the target question and answer information corresponding to the current query information, and the target question and answer information is question and answer information stored in the first database that matches the current query information, and the first database is used to store at least one question and answer information and a target cutting image generated by any of the above methods.

[0134] It should be noted that the first display module 902 and the second display module 904 correspond to steps S502 to S504 in the above embodiment, and the examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also be run in the server 10 provided in the above embodiment as part of the device.

[0135] According to an embodiment of the present application, a data processing device for implementing the above data processing method is also provided. Fig.10 is a structural block diagram of another data processing device according to the present application, such as Fig.10 As shown, the device includes: an information calling module 1002, an information matching module 1004, a reply generating module 1006 and an information output module 1008.

[0136] Among them, the information calling module 1002 is used to obtain the current query information by calling the first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter includes the current query information, and the current query information is used to inquire about the attribute characteristics of the target object; the information matching module 1004 is used to obtain target question and answer information matching the current query information from the first database based on the current query information, wherein the first database is used to store at least one question and answer information and a target cutting image generated by any of the above methods; the reply generation module 1006 is used to generate current reply information corresponding to the current query information based on at least the target question and answer information; the information output module 1008 is used to output the current reply information by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the current reply information.

[0137] It should be noted that the information calling module 1002, information matching module 1004, reply generating module 1006 and information output module 1008 correspond to steps S602 to S608 in the above embodiment, and the four modules and corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also be run in the server 10 provided in the above embodiment as part of the device.

[0138] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in the above embodiments, as well as the application scenario and implementation process, but is not limited to the scheme provided in the above embodiments.

[0139] The embodiment of the present application may provide an electronic device, which may be any electronic device in a group of electronic devices. Optionally, in this embodiment, the electronic device may also be replaced by a terminal device such as a mobile terminal.

[0140] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0141] In this embodiment, the electronic device can execute the program code in the method.

[0142] Optionally, Fig.11 is a structural block diagram of an electronic device according to the present application. Fig.11 As shown, the electronic device A may include: one or more (only one is shown in the figure) processors 1102, a memory 1104, a storage controller, and a peripheral interface, wherein the peripheral interface is connected to a radio frequency module, an audio module, and a display.

[0143] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implementing the methods in the above embodiments. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0144] The processor can call the information and application stored in the memory through the transmission device to execute the following steps: obtain a target image corresponding to a preset object, wherein the target image includes an image of the preset object and text information used to describe the attribute characteristics of the preset object; perform optical character recognition on the target image to obtain the text information contained in the target image; extract information from the target image based on the text information to generate at least one question and answer information, wherein the inquiry information in at least one question and answer information is used to inquire about the attribute characteristics of the preset object, and the reply information in at least one question and answer information is used to reply to the inquiry information; store at least one question and answer information and the target image in a first database.

[0145] Those skilled in the art will appreciate that the structure shown in the figure is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), a PAD, and other terminal devices. Fig.11 It does not limit the structure of the above electronic device. For example, the electronic device A may also include Fig.11 More or fewer components (such as network interfaces, display devices, etc.) shown in the figure, or having Fig.11 Different configurations are shown.

[0146] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0147] The embodiment of the present application further provides a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the method provided in the above embodiment.

[0148] Optionally, in this embodiment, the above storage medium may be located in any electronic device in a group of electronic devices in a computer network, or in any mobile terminal in a group of mobile terminals.

[0149] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: obtaining a target image corresponding to a preset object, wherein the target image includes an image of the preset object and text information for describing attribute characteristics of the preset object; performing optical character recognition on the target image to obtain text information contained in the target image; extracting information from the target image based on the text information to generate at least one question and answer information, wherein the inquiry information in at least one question and answer information is used to inquire about the attribute characteristics of the preset object, and the reply information in at least one question and answer information is used to reply to the inquiry information; storing at least one question and answer information and the target image in a first database.

[0150] The embodiment of the present application further provides a computer program product. Optionally, in this embodiment, the computer program product may include a computer program, and the computer program implements the method provided in the embodiment when executed by a processor.

[0151] The embodiments of the present application also provide a computer program product. Optionally, the computer program product may include a non-volatile computer-readable storage medium, which may be used to store a computer program, and when the computer program is executed by a processor, the method provided in the embodiments is implemented.

[0152] The embodiment of the present application further provides a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, the method provided in the above embodiment is implemented.

[0153] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0154] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0155] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0156] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0157] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0158] The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data processing method, characterized in that: include: Acquire a target image corresponding to a preset object, wherein the target image includes an image of the preset object and text information for describing attribute characteristics of the preset object; Recognize the target image to obtain the text information contained in the target image; The text information and the target image are input into a multimodal large model, and the multimodal large model is used to extract information from the target image to generate at least one question and answer message, wherein the inquiry information in the at least one question and answer message is used to inquire about the attribute characteristics of the preset object, and the reply information in the at least one question and answer message is used to reply to the inquiry information.

2. The method according to claim 1, characterized in that The method further comprises: Cutting the target image based on the text information to obtain a target cut image; The at least one question-and-answer information and the target cutting image are stored in a first database.

3. The method according to claim 2, characterized in that The method further comprises: Cutting the target image according to a preset size to obtain an initial cut image; The identifying the target image to obtain the text information contained in the target image includes: performing optical character recognition on the initial cutting image to obtain the text information; The step of cutting the target image based on the text information to obtain a target cut image includes: The target image is cut based on the text information to obtain the target cut image.

4. The method according to claim 3, characterized in that The method further comprises: Converting the text information into a text image; The text image, the initial cutting image and the target cutting image are stored in a second database.

5. The method according to claim 2, characterized in that: After cutting the target image based on the text information, the method further includes: Determine whether the target image is successfully cut, and obtain a cutting result of the target image; The segmentation result of the target image is stored in the first database.

6. The method according to claim 5, characterized in that When it is determined that the target image is successfully cut, the method further includes: The current number of cutting images corresponding to the preset object stored in the third database is updated to obtain the target number of cutting images corresponding to the preset object, wherein the current number of cutting images is used to represent the number of historical cutting images corresponding to the preset object stored in the first database.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises one of the following: In the case of receiving target query information for inquiring about the attribute characteristics of the preset object, determining whether the target query information is preset query information, wherein the preset query information is used to represent the query information for inquiring about the attribute characteristics of the preset object received for the first time; In a case where the target inquiry information is not the preset inquiry information, determining whether the at least one question-and-answer information is stored in the first database; When the target query information is the preset query information, or the at least one question and answer information is not stored in the first database, the steps of performing optical character recognition on the target image, extracting the target image based on the text information, and storing the at least one question and answer information and the target image in the first database are performed.

8. The method according to any one of claims 1 to 6, characterized in that: The method further comprises one of the following: In case of receiving change information for changing the original image of the preset object into the target image, determining whether the original image is stored in the first database; In the case where the original image is stored in the first database, the steps of performing optical character recognition on the target image, extracting the target image based on the text information, and storing the at least one question and answer information and the target image in the first database are performed.

9. A data processing method, characterized in that: include: Accepting current query information sent by the client, wherein the current query information is used to query the attribute characteristics of the target object; Based on the current query information, obtaining target question and answer information matching the current query information from a first database, wherein the first database is used to store at least one question and answer information and a target cutting image generated by the method according to any one of claims 1 to 8; At least based on the target question and answer information, generate current reply information corresponding to the current inquiry information; Send the current reply information to the client.

10. The method according to claim 9, characterized in that The generating, based at least on the target question and answer information, current reply information corresponding to the current inquiry information comprises: Based on the current query information, obtaining multiple description information corresponding to the current query information from multiple data sources, wherein different description information is used to represent information from different data sources that describes the attribute characteristics of the target object; Merging the target question and answer information and the plurality of description information to obtain a merged result; The merged result is input into the multi-source question-answering big model, and the current reply information output by the multi-source question-answering big model is obtained.

11. The method according to claim 9 or 10, characterized in that: In the case where the target question and answer information is not obtained from the first database, the method further includes: Acquire a preset image corresponding to the target object, wherein the preset image includes an image of the target object and text information for describing attribute characteristics of the target object; Identify the current query information and the preset image to determine the attribute characteristics of the target object; Acquire multiple descriptive information matching the attribute characteristics of the target object from multiple data sources; The current reply information is generated based on the multiple description information.

12. A data processing method, characterized in that: include: In response to an input instruction acting on the operation interface, current query information is displayed on the operation interface, wherein the current query information is used to query the attribute characteristics of the target object; In response to an inquiry instruction applied to the operation interface, current reply information corresponding to the current inquiry information is displayed on the operation interface, wherein the current reply information is information generated based at least on target question and answer information corresponding to the current inquiry information, and the target question and answer information is question and answer information matching the current inquiry information stored in a first database, and the first database is used to store at least one question and answer information and a target cutting image generated by the method described in any one of claims 1 to 8.

13. A data processing method, characterized in that: include: Acquire current query information by calling a first interface, wherein the first interface includes a first parameter, a parameter value of the first parameter includes the current query information, and the current query information is used to query the attribute characteristics of the target object; Based on the current query information, obtaining target question and answer information matching the current query information from a first database, wherein the first database is used to store at least one question and answer information and a target cutting image generated by the method according to any one of claims 1 to 8; At least based on the target question and answer information, generate current reply information corresponding to the current inquiry information; The current reply information is output by calling a second interface, wherein the second interface includes a second parameter, and a parameter value of the second parameter includes the current reply information.

14. An electronic device, characterized in that: include: A memory storing an executable program; A processor, configured to run the program, wherein the program executes the method according to any one of claims 1 to 13 when running.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 13.

16. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 13.

Images