Information processing method and device, electronic equipment and storage medium

Abstract

The invention provides an information processing method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence, in particular to the technical field of computer vision, large models and deep learning. According to the specific implementation scheme, the method comprises the steps that prompt information is analyzed, and multi-level constraint information is determined; identifying a to-be-processed target image, and obtaining a visual vector corresponding to the target image; based on the visual vector and first-level constraint information in the multi-level constraint information, determining first probability distribution corresponding to each candidate word in the word list; based on other levels of constraint information in the multi-level constraint information, updating the first probability distribution to obtain target probability distribution; and determining a first output character based on the target probability distribution, and returning to execute the operation of determining the first probability distribution corresponding to each candidate word based on the first output character, the visual vector and first-level constraint information in the multi-level constraint information until the generated output character is an end character. The deep layer capability of the model can be stimulated, and self-driving work is disassembled.

Description

Information processing methods, devices, electronic equipment and storage media Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to the fields of computer vision, large models, and deep learning technology, specifically to an information processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Cue words serve as a bridge connecting user needs with the generation capabilities of large models such as the Vision-Language Model (VLM), directly determining the accuracy, relevance, and quality of visual output. Therefore, optimizing cue words to improve the model's visual processing capabilities has become crucial for model development. Summary of the Invention

[0003] This disclosure aims to at least partially address one of the technical problems in the related art.

[0004] Therefore, the purpose of this disclosure is to propose an information processing method, apparatus, electronic device and storage medium that injects the context of the image into the model through structured and multi-dimensional prompts, stimulates the model's ability to process visual information in the image in depth, decomposes the self-driven work, and improves the quality of the model's processing results.

[0005] According to a first aspect of this disclosure, an information processing method is provided, comprising: parsing prompt information to determine multi-level constraint information; recognizing a target image to be processed to obtain a visual vector corresponding to the target image; determining a first probability distribution corresponding to each candidate word in a word list based on the visual vector and the first-level constraint information in the multi-level constraint information; updating the first probability distribution based on other-level constraint information in the multi-level constraint information to obtain a target probability distribution; determining a first output character based on the target probability distribution, and returning to perform the operation of determining the first probability distribution corresponding to each candidate word based on the first output character, the visual vector, and the first-level constraint information in the multi-level constraint information, until the generated output character is an end character.

[0006] According to a second aspect of this disclosure, an information processing apparatus is provided, comprising: a first determining module, configured to parse prompt information and determine multi-level constraint information; a recognition module, configured to recognize a target image to be processed and obtain a visual vector corresponding to the target image; a second determining module, configured to determine a first probability distribution corresponding to each candidate word in a word list based on the visual vector and the first-level constraint information in the multi-level constraint information; an updating module, configured to update the first probability distribution based on other-level constraint information in the multi-level constraint information to obtain a target probability distribution; and an output module, configured to determine a first output character based on the target probability distribution, and, based on the first output character, the visual vector, and the first-level constraint information in the multi-level constraint information, return to the operation of determining the first probability distribution corresponding to each candidate word, until the generated output character is an end character.

[0007] According to a third aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the information processing method as described in the first aspect.

[0008] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided that stores computer instructions for causing the computer to perform the information processing method as described in the first aspect.

[0009] According to a fifth aspect of this disclosure, a computer program product is provided, including computer instructions that, when executed by a processor, implement the steps of the information processing method as described in the first aspect.

[0010] The information processing method, apparatus, electronic device, and storage medium provided in this disclosure have the following beneficial effects: by updating the probability distribution of candidate words in the image to be processed level by level based on the multi-level constraints in the prompt information, and then outputting characters one by one based on the final probability distribution, the processing results of the model on the image output can be obtained. This can stimulate the model's ability to process visual information in the image in a deeper way, decompose the self-driven work, and improve the quality of the model's processing results.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The above and / or additional aspects and advantages of this disclosure will become apparent and readily understood from the following description of embodiments taken in conjunction with the accompanying drawings, which are provided for a better understanding of the present invention and are not intended to limit the scope of this disclosure. In the drawings: FIG1 is a flowchart illustrating an information processing method according to an embodiment of this disclosure; FIG2 is a flowchart illustrating an information processing method according to another embodiment of this disclosure; FIG3 is a flowchart illustrating an information processing method according to another embodiment of this disclosure; FIG4 is a structural schematic diagram of an information processing apparatus according to an embodiment of this disclosure; FIG5 shows a block diagram of an exemplary electronic device suitable for implementing embodiments of this disclosure. Detailed Implementation

[0013] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0014] The embodiments disclosed herein relate to the fields of artificial intelligence technology, such as computer vision, large models, and deep learning.

[0015] Artificial Intelligence (AI) is a new technological science that studies, develops, and applies theories, methods, technologies, and application systems to simulate, extend, and expand human intelligence.

[0016] Computer vision refers to machine vision that uses cameras and computers to identify, track, and measure targets instead of human eyes, and further processes the images to make them more suitable for human observation or transmission to instruments for detection.

[0017] Large models refer to a class of artificial intelligence models with a large number of parameters built from artificial neural networks. The main types include large language models, large vision models, multimodal large models, and large basic science models. A large vision model (VLM) achieves accurate understanding of image content and transforms it into natural and fluent language descriptions by aligning visual understanding and language models. Its capabilities include, but are not limited to, visual question answering, information extraction, and information summarization.

[0018] Deep learning learns the inherent patterns and hierarchical representations of sample data. The information gained during this learning process greatly aids in interpreting data such as text, images, and sound. The ultimate goal of deep learning is to enable machines to possess analytical and learning capabilities similar to humans, allowing them to recognize data such as text, images, and sound.

[0019] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0020] This disclosure also involves prompts. Prompts are a way to guide or instruct large models to complete tasks and achieve the desired results through natural language. Prompt engineering is a systematic technique for optimizing prompt performance.

[0021] The information processing method, apparatus, electronic device, and storage medium of this disclosure are described below with reference to the accompanying drawings.

[0022] It should be noted that the execution subject of the information processing method in this embodiment is an information processing device, which can be implemented by software and / or hardware. This device can be configured in an electronic device, which may include, but is not limited to, a terminal or a server. This embodiment uses the example of an information processing device configured in an information processing system for illustration.

[0023] Figure 1 is a schematic flowchart of an information processing method according to an embodiment of the present disclosure.

[0024] As shown in Figure 1, the information processing method includes: S101: parsing the prompt information to determine multi-level constraint information.

[0025] In this embodiment of the disclosure, the prompt information can be generated based on a general prompt word template, after filling in information such as the scene of the image to be processed, the expert role, and the user intent, and is used to instruct the model to process the image and output the results.

[0026] In this embodiment of the disclosure, the prompt information can be parsed by identifying separators in the prompt information and dividing it into layers according to the semantics of the prompt information, thereby extracting multi-level constraint information from the prompt information.

[0027] For example, the multi-level constraint information obtained from parsing the prompt message "This is a bill of lading, and you are the document reviewer: 1. Please describe your work process, 2. My requirement is: extract its information, 3. Please check the conclusion and output in key-value pair format" can include "This is a bill of lading, and you are the document reviewer", "Please describe your work process", "My requirement is: extract its information", "Please check the conclusion", and "output in key-value pair format".

[0028] It should be noted that in different embodiments of this disclosure, the number of levels, precision, etc. of multi-level constraint information may be different, and this disclosure does not limit this.

[0029] S102: Recognize the target image to be processed and obtain the visual vector corresponding to the target image.

[0030] In this embodiment of the disclosure, the visual vector can be a string of high-dimensional numerical values ​​generated after the large model recognizes the target image and extracts the features in the image, that is, the mathematical representation of the image content in the model feature space.

[0031] It should be noted that the information processing method provided in this disclosure can be applied to image processing in different scenarios. For example, the target image can be official documents, archives, financial receipts, etc. in corporate office and government scenarios; it can be car insurance accident pictures, certificates, etc. in financial insurance scenarios; or it can be medical images, academic papers, etc. in medical research scenarios. It has a wider range of applications and a certain degree of universality.

[0032] S103: Based on the visual vector and the first-level constraint information in the multi-level constraint information, determine the first probability distribution corresponding to each candidate word in the vocabulary.

[0033] The first-level constraint information can be the constraint information that the large model processes first according to the order of processing prompt information (such as from left to right or from top to bottom), or it can be the constraint information that has the highest impact on business scenarios and user intentions among the multi-level constraint information.

[0034] In this embodiment of the disclosure, the candidate words in the vocabulary may be constructed based on the historical training of a large model, or may be obtained by text recognition of the target image, or may be determined by other methods, and this disclosure does not limit them.

[0035] In this embodiment of the disclosure, a first probability distribution can be determined based on the matching relationship between each candidate word and the visual vector and the first-level constraint information. The higher the degree of matching, the higher the probability corresponding to the candidate word. For example, if the first-level constraint information is "This is a bill of lading, and you are the document reviewer," the probability corresponding to the candidate word "bill of lading number" in the first probability distribution will be greater than the probability corresponding to the candidate word "volume."

[0036] S104: Based on the constraint information of other levels in the multi-level constraint information, update the first probability distribution to obtain the target probability distribution.

[0037] In this embodiment of the disclosure, the probability distribution after each update can be updated step by step according to the matching relationship between other level constraint information and each candidate word. The probability distribution obtained after completing the update operation of all level constraint information is determined as the target probability distribution.

[0038] It should be noted that, in this embodiment of the disclosure, when other level constraint information is "please describe your work process" or other information that can instruct the large model to demonstrate the thought process, the large model can determine the probability of each candidate word and update the probability distribution based on historical training data related to the target image.

[0039] S105: Determine the first output character based on the target probability distribution, and based on the first output character, the visual vector, and the first-level constraint information in the multi-level constraint information, return to perform the operation of determining the first probability distribution corresponding to each candidate word, until the generated output character is the end character.

[0040] In this embodiment of the disclosure, the candidate word with the highest probability in the target probability distribution can be determined as the first output character. Then, the operation of determining the first probability distribution corresponding to each candidate word can be returned to continue determining the next output character until the output character is the end character, so as to obtain the complete processing result of the large model outputting the target image.

[0041] In this embodiment, the probability distribution of candidate words in the image to be processed is updated step by step based on the multi-level constraints in the prompt information. Then, the characters are output one by one based on the final probability distribution to obtain the processing results of the model on the image output. This can stimulate the model's ability to process visual information in the image in depth, decompose the self-driven work, and improve the quality of the model's processing results.

[0042] Figure 2 is a flowchart illustrating an information processing method according to another embodiment of this disclosure.

[0043] As shown in Figure 2, the information processing method includes: S201: parsing the target image to determine at least one candidate content associated with each of the multiple prompts in the preset prompt template.

[0044] Among them, multiple prompts can correspond to different dimensions, such as scenario type, the type of expert role played by the model in processing information, and user purpose and intent.

[0045] In this embodiment of the disclosure, only one prompt word template needs to be preset. This prompt word template is universal in different application scenarios. By simply filling in the content of a limited number of prompt items in the prompt word template, it can be applied to image processing in any scenario, greatly eliminating the difference in effect caused by reliance on professionalism, experience, and usage habits in different scenarios.

[0046] In this embodiment of the disclosure, by parsing the content of the target image, it is possible to predict one or more business scenarios to which the target image may belong, the type of expert role that can process the target image, and what information can be obtained from the target image, etc., as at least one candidate content associated with each prompt item.

[0047] For example, a preset prompt template could be: "This is scenario {A}, you are a {B}, describe your work process (or methodology) based on what you see, my request is {C}, please check the conclusion." Here, {} mark the prompt items in the prompt template. For a document image, candidate content for prompt item A could be a bill of lading, invoice, etc., while candidate content for prompt item B could be the document reviewer, finance personnel, etc.

[0048] S202: Display each prompt item and at least one candidate item together.

[0049] In this embodiment of the disclosure, at least one candidate content associated with each prompt item can be displayed by means of drop-down selection boxes, placeholder prompts at the prompt items, etc., and there is no limitation thereto.

[0050] S203: Based on user actions, determine the target content corresponding to each prompt item.

[0051] In this embodiment of the disclosure, the user can perform operations such as selecting candidate content or inputting at the prompt, so that the large model can determine the target content corresponding to each prompt.

[0052] In some embodiments, the user's selection for each prompt can be determined based on the user's actions, and the correlation between the selections corresponding to multiple prompts can be determined.

[0053] In this embodiment of the disclosure, the relevance between the selected content corresponding to multiple prompts can be determined by calculating the vector similarity or semantic association of the selected content, so as to avoid the problem of poor information processing accuracy caused by the unrelated content of multiple prompts selected by the user.

[0054] Then, if the correlation between the selected content corresponding to at least two prompts is less than a threshold, clarification information can be generated and displayed based on the selected content corresponding to at least two prompts.

[0055] The threshold can be determined based on experience, etc., and is the minimum correlation value that can ensure a high degree of accuracy in the processing results.

[0056] For example, if the scene prompt is medical image and the role prompt is after-sales personnel, the correlation between medical image and after-sales personnel is relatively low. If the prompt information is built based on the selected content, it may cause the model information processing failure and the result accuracy to be poor. Therefore, clarification information can be generated and displayed based on the selected content corresponding to these two prompts to remind the user to modify the prompts with a correlation of less than the threshold.

[0057] Subsequently, upon receiving user input, the target content corresponding to each of the at least two prompts is determined based on the user input.

[0058] In this embodiment of the disclosure, by calculating the relevance of the content selected by the user and modifying the content of the prompts that have a relevance below a certain threshold, the accuracy of the prompts can be guaranteed, thereby ensuring the quality and efficiency of information processing.

[0059] S204: Generate prompt information based on the target content and prompt word template.

[0060] In this embodiment of the disclosure, after determining the target content corresponding to each prompt item, each target content can be filled into the corresponding prompt item position in the prompt word template to obtain prompt information.

[0061] In this embodiment, the positions of each prompt item in the prompt word template are intelligently determined by parsing the image and matching the fillable candidate content of the image. Then, prompt information is generated based on the candidate content selected by the user. This can improve the efficiency of prompt word determination, effectively reduce the professional requirements of users in different scenarios, and has greater versatility.

[0062] Understandably, in many scenarios, users are not experts but need to obtain professional answers using large models. Pre-trained VLM models have sufficient information processing capabilities. Compared to further training with data from different scenarios or designing targeted prompts, it is more convenient and faster to allow the large model to complete the design, execution, and response of the scenario itself with the capabilities, style, and behavioral logic of a scenario expert, thus stimulating the model's information processing capabilities.

[0063] This disclosure uses pre-set prompt templates to give large models an expert persona, allowing them to describe their workflow in a given scenario (e.g., after observing an image, the model, as the expert, immediately thinks of what it should do and what steps it should take). The model can then verify its work results, optimizing the answer based on business understanding rather than simple extraction. This allows non-professional users to better complete scenario tasks, while also improving the efficiency of professional users by eliminating the need to customize prompts for different visual images in different scenarios.

[0064] Figure 3 is a flowchart illustrating an information processing method according to another embodiment of this disclosure.

[0065] As shown in Figure 3, the information processing method includes: S301: parsing the prompt information to determine multi-level constraint information.

[0066] S302: Recognize the target image to be processed and obtain the visual vector corresponding to the target image.

[0067] The descriptions of S301 and S302 above can be found in the above embodiments, and will not be repeated here.

[0068] S303: Determine the current scenario and the target user's intent.

[0069] In this embodiment of the disclosure, the scene to which the target image is currently being processed and the user's intent to process the target image can be determined by user input, or the target image can be automatically parsed to determine candidate scenes and intents, and then the user can select the current scene and the target user intent, etc. This disclosure does not limit this.

[0070] S304: Based on the influence of preset constraint information on different scenarios and intentions, determine the constraint information with the greatest influence under the current scenario and target user intention.

[0071] In this embodiment of the disclosure, multiple constraint information corresponding to the preset prompt word template can be determined. Since the degree of influence of each constraint information on the model output may be different under different scenarios and intentions, the higher the degree of influence, the more conducive it is for the model to quickly lock the better output content. Therefore, the constraint information with the greatest influence under the current scenario and the target user's intention can be determined first.

[0072] S305: The constraint with the greatest impact is identified as the first-level constraint.

[0073] In this embodiment, the constraint information with the greatest influence is determined as the first-level constraint information, which is used to initially determine the first probability distribution of candidate word screening, and can more quickly determine which candidate words in the word list are more in line with the current scenario and user intent.

[0074] For example, if we predetermine the influence of constraint information 1 on scenario 1 and intent 1 as 0.6, the influence of constraint information 2 as 0.3, and the influence of constraint information 3 as 0.1; and for scenario 1 and intent 2: the influence of constraint information 1 as 0.4, the influence of constraint information 2 as 0.5, and the influence of constraint information 3 as 0.1, then in scenario 1 and intent 1, since 0.6 > 0.3 > 0.1, constraint information 1 is the first-level constraint information; while in scenario 1 and intent 2, since 0.5 > 0.4 > 0.1, constraint information 2 is the first-level constraint information.

[0075] In this embodiment, by selecting the constraint information with the greatest influence as the first-level constraint information for determining the probability distribution based on the influence of different constraint information under the current scenario and the target user's intent, the efficiency and accuracy of probability distribution determination can be improved, which is beneficial to reducing the time consumption of probability distribution updates.

[0076] S306: Based on the visual vector and the first-level constraint information in the multi-level constraint information, determine the first probability distribution corresponding to each candidate word in the vocabulary.

[0077] For a detailed description of S306, please refer to the above embodiments, which will not be repeated here.

[0078] S307: Based on the first probability distribution, filter the candidate words to determine the filtered candidate words and the corresponding second probability distribution.

[0079] In this embodiment of the disclosure, after obtaining the first probability distribution, candidate words with particularly low probabilities (such as probabilities less than 0.2) in the first probability distribution can be deleted to obtain filtered candidate words. The second probability distribution is determined by the probabilities corresponding to these filtered candidate words.

[0080] S308: Based on the constraint information of other levels in the multi-level constraint information, the second probability distribution is updated to obtain the target probability distribution.

[0081] In this embodiment of the disclosure, the second probability distribution can be updated by increasing or decreasing the probability corresponding to each candidate word in the second probability distribution based on other constraint information at all levels.

[0082] It should be noted that when there are multiple constraints at other levels, the probability distribution can be updated sequentially in descending order of the influence of each constraint level. Furthermore, after each update of the probability distribution, the candidate words can be filtered again.

[0083] In this embodiment, a relatively accurate first probability distribution is determined based on the constraint with the greatest impact. This allows for the screening of candidate words based on the relatively accurate first probability distribution. In the later probability update, only a portion of the word list needs to be considered, which maximizes the amount of data processed for probability distribution update and improves processing efficiency.

[0084] S309: Determine the first output character based on the target probability distribution, and based on the first output character, the visual vector, and the first-level constraint information in the multi-level constraint information, return to perform the operation of determining the first probability distribution corresponding to each candidate word, until the generated output character is the end character.

[0085] For a detailed description of S309, please refer to the above embodiments, which will not be repeated here.

[0086] In some embodiments, where the multi-level constraint information includes constraint information associated with the output, the generated content can be adjusted based on the constraint information associated with the output to obtain the target output result.

[0087] In this embodiment of the disclosure, the constraint information associated with the output can be the format of the constraint output, such as key-value (KV) format, table format, etc.

[0088] For example, in the prompt message "This is a bill of lading for ocean freight, and you are the document reviewer: 1. Please describe your work process, 2. My requirement is: extract its information, 3. Please check the conclusion," the multi-level constraint information includes the constraint information "output is in key-value pair format" associated with the output. In this case, the target output of the large model should be in key-value pair format.

[0089] It should be noted that if the multi-level constraint information does not contain constraint information related to the output, the generated content can be adjusted by default according to the style related to the scenario and expert role in the constraint information to obtain the target output result.

[0090] In this embodiment, by constraining the output of the large model in the prompt message, the information processing results can be made more in line with user needs, improving the quality of the processing results and optimizing the user experience.

[0091] Figure 4 is a schematic diagram of the structure of an information processing device according to an embodiment of this disclosure.

[0092] As shown in Figure 4, the information processing device 40 includes: a first determining module 401, used to parse the prompt information and determine multi-level constraint information; a recognition module 402, used to recognize the target image to be processed and obtain the visual vector corresponding to the target image; a second determining module 403, used to determine the first probability distribution corresponding to each candidate word in the word list based on the visual vector and the first-level constraint information in the multi-level constraint information; an updating module 404, used to update the first probability distribution based on other levels of constraint information in the multi-level constraint information to obtain the target probability distribution; and an output module 405, used to determine the first output character based on the target probability distribution, and based on the first output character, the visual vector, and the first-level constraint information in the multi-level constraint information, return to the operation of determining the first probability distribution corresponding to each candidate word until the generated output character is the end character.

[0093] In some embodiments of this disclosure, the first determining module 401 may also be used to: parse the target image to determine at least one candidate content associated with each of the multiple prompts in the preset prompt word template, wherein the multiple prompts correspond to different dimensions; display each prompt and at least one candidate content in association; determine the target content corresponding to each prompt based on user operation; and generate prompt information based on the target content and the prompt word template.

[0094] In some embodiments of this disclosure, the first determining module 401 may be specifically used to: determine the selection content determined by the user for each prompt item based on user operation; determine the correlation between the selection content corresponding to multiple prompt items; generate and display clarification information based on the selection content corresponding to at least two prompt items when the correlation between the selection content corresponding to at least two prompt items is less than a threshold; and determine the target content corresponding to at least two prompt items based on user input when user input is received.

[0095] In some embodiments of this disclosure, the second determining module 403 may also be used to: determine the current scenario and the target user's intent; determine the constraint information with the greatest influence under the current scenario and the target user's intent based on the influence of preset constraint information on different scenarios and intents; and determine the constraint information with the greatest influence as the first-level constraint information.

[0096] In some embodiments of this disclosure, the update module 404 may be specifically used to: filter candidate words based on a first probability distribution, determine the filtered candidate words and the second probability distribution corresponding to the filtered candidate words; update the second probability distribution based on other level constraint information in the multi-level constraint information to obtain the target probability distribution.

[0097] In some embodiments of this disclosure, the information processing device 40 may further include a processing module, which may be used to: adjust the generated content based on the constraint information associated with the output when the multi-level constraint information includes constraint information associated with the output, so as to obtain the target output result.

[0098] It should be noted that the foregoing explanation of the information processing method also applies to the information processing device of this embodiment, and will not be repeated here.

[0099] In this embodiment, the probability distribution of candidate words in the image to be processed is updated step by step based on the multi-level constraints in the prompt information. Then, the characters are output one by one based on the final probability distribution to obtain the processing results of the model on the image output. This can stimulate the model's ability to process visual information in the image in depth, decompose the self-driven work, and improve the quality of the model's processing results.

[0100] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0101] Figure 5 illustrates a schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0102] As shown in Figure 5, device 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 502 or a computer program loaded into random access memory (RAM) 503 from storage unit 508. RAM 503 can also store various programs and data required for the operation of device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0103] Multiple components in device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0104] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as information processing methods. For example, in some embodiments, the information processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the information processing method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform information processing methods by any other suitable means (e.g., by means of firmware).

[0105] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0106] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0107] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0108] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0109] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0110] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system that addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0111] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0112] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified. In the description of this disclosure, the words "if" and "suppose" as used may be interpreted as "when," "when," "in response to determination," or "in the circumstances."

[0113] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An information processing method, comprising: The prompt information is parsed to determine the multi-level constraint information; The target image to be processed is identified, and the visual vector corresponding to the target image is obtained; Based on the visual vector and the first-level constraint information in the multi-level constraint information, a first probability distribution corresponding to each candidate word in the vocabulary is determined; based on the other-level constraint information in the multi-level constraint information, the first probability distribution is updated to obtain a target probability distribution; based on the target probability distribution, a first output character is determined, and based on the first output character, the visual vector, and the first-level constraint information in the multi-level constraint information, the operation of determining the first probability distribution corresponding to each candidate word is returned until the generated output character is an end character.

2. The method as described in claim 1, wherein, Before parsing the prompt information and determining the multi-level constraint information, the method further includes: parsing the target image to determine at least one candidate content associated with each prompt item in a preset prompt word template, wherein the multiple prompt items correspond to different dimensions; displaying each prompt item and at least one candidate content in association; determining the target content corresponding to each prompt item based on user operation; and generating the prompt information based on the target content and the prompt word template.

3. The method as described in claim 2, wherein, The step of determining the target content corresponding to each prompt item based on user operation includes: determining the selection content determined by the user for each prompt item based on the user operation; determining the correlation between the selection content corresponding to the plurality of prompt items respectively; generating and displaying clarification information based on the selection content corresponding to the at least two prompt items when the correlation between the selection content corresponding to at least two prompt items is less than a threshold; and determining the target content corresponding to the at least two prompt items based on the user input when user input is received.

4. The method as described in any one of claims 1-3, wherein, Before determining the first probability distribution corresponding to each candidate word in the vocabulary based on the visual vector and the first-level constraint information in the multi-level constraint information, the method further includes: determining the current scene and the target user intent; determining the constraint information with the greatest influence under the current scene and the target user intent based on the influence degree of the preset constraint information on different scenes and intents; and determining the constraint information with the greatest influence as the first-level constraint information.

5. The method of claim 4, wherein, The step of updating the first probability distribution based on other level constraint information in the multi-level constraint information to obtain the target probability distribution includes: filtering the candidate words based on the first probability distribution to determine the filtered candidate words and the second probability distribution corresponding to the filtered candidate words; and updating the second probability distribution based on other level constraint information in the multi-level constraint information to obtain the target probability distribution.

6. The method of claim 4, wherein, The method further includes: when the multi-level constraint information includes constraint information associated with the output, adjusting the generated content based on the constraint information associated with the output to obtain the target output result.

7. An information processing apparatus, comprising: The first determination module is used to parse the prompt information and determine the multi-level constraint information; The recognition module is used to recognize the target image to be processed and obtain the visual vector corresponding to the target image; The second determining module is used to determine the first probability distribution corresponding to each candidate word in the word list based on the visual vector and the first-level constraint information in the multi-level constraint information; The update module is used to update the first probability distribution based on other level constraint information in the multi-level constraint information to obtain the target probability distribution; the output module is used to determine the first output character based on the target probability distribution, and based on the first output character, the visual vector and the first level constraint information in the multi-level constraint information, return to the operation of determining the first probability distribution corresponding to each candidate word, until the generated output character is the end character.

8. An electronic device, comprising: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the information processing method according to any one of claims 1-6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, in, The computer instructions are used to cause the computer to perform the information processing method according to any one of claims 1-6.

10. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the steps of the information processing method according to any one of claims 1-6.

Images