A form filling method and device based on virtual object

Abstract

The present invention provides a form filling method and device based on virtual objects, and relates to the field of artificial intelligence technology. The method includes: displaying a form to be filled in; if a trigger operation for a virtual object is detected, determining the trigger type of the trigger operation and displaying the virtual object while displaying the form to be filled in; based on the operation type of the trigger operation, determining the target form item from multiple form items; using a specified template and a target form item, generating query information and controlling the output of the virtual object; in response to feedback information for the query information, using the feedback information to control the virtual object to perform an auxiliary filling action for the target form item. The present invention can provide a virtual object to assist the user in filling in the form during the form filling process, which is convenient and quick, and can effectively improve the efficiency and accuracy of data collection.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a form filling method and device based on virtual objects. Background Art

[0002] In the current era of rapid development of informatization, forms, as the core tool for data collection and processing, are widely used in various online service platforms and systems. As business needs become more complex, the design of form items in forms also tends to be more refined and complex to ensure the accuracy and comprehensiveness of data collection.

[0003] However, these form items are not only diverse in type, but also often have complex association logics between each other. When filling out these form items, users usually find it difficult to accurately understand the meaning of the form items, and need to ask others or use the Internet to search before filling out. This method is time-consuming and laborious, and there is no guarantee that the filling method provided by others or the Internet is correct, making it difficult for users to fill out the corresponding content efficiently and accurately, resulting in low efficiency and accuracy of data collection. Summary of the invention

[0004] In view of the above problems, the purpose of the present invention is to provide a form filling method and device based on virtual objects, which can assist users in filling out forms and improve the accuracy and efficiency of data collection.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0006] On the one hand, an embodiment of the present invention provides a vision-based question-answering processing method, comprising:

[0007] Displaying a form to be filled in, wherein the form to be filled in includes a plurality of form items to be filled in;

[0008] If a trigger operation for a virtual object is detected, determining a trigger type of the trigger operation, and displaying the virtual object while displaying the form to be filled in;

[0009] Based on the trigger type of the trigger operation, determining a target form item from the multiple form items, the target form item being the form item that currently needs to be assisted in filling out;

[0010] Generate query information using a specified template and the target form item and control the virtual object to output the query information;

[0011] In response to feedback information for the inquiry information, the virtual object is controlled by using the feedback information to perform an auxiliary filling action for the target form item, wherein the auxiliary filling action includes outputting auxiliary filling information or filling in the target form item.

[0012] Optionally, the trigger type includes an active type, and if a trigger operation for a virtual object is detected, determining the trigger type of the trigger operation, and displaying the virtual object while displaying the form to be filled in, comprises:

[0013] If a triggering voice is detected, the triggering voice is analyzed and processed to obtain a triggering intention;

[0014] If the triggering intention is an intention to seek help, determining that an active type of triggering operation is detected;

[0015] If the active type of trigger operation is detected, the virtual object and the form to be filled out are displayed in a first arrangement.

[0016] Optionally, the determining a target form item from the multiple form items based on the trigger type of the trigger operation includes:

[0017] Converting the trigger voice into a trigger text;

[0018] Based on the trigger text and the description information corresponding to each of the form items, detecting a candidate form item corresponding to the trigger text from the multiple form items;

[0019] If the candidate form item is detected, determining the candidate form item as a target form item;

[0020] If the candidate form item is not detected, based on the mouse event, an indicated form item is determined from the multiple form items, and the indicated form item is used as the target form item, where the indicated form item is the form item indicated by the cursor.

[0021] Optionally, the description information includes a specific description and a filling explanation, and the detecting, from the multiple form items, a candidate form item corresponding to the trigger text based on the trigger text and the description information corresponding to each of the form items includes:

[0022] For each of the form items, calculating a first matching degree between the specific description and the trigger text;

[0023] Calculating a second matching degree between the filling explanation and the trigger text;

[0024] The first matching degree and the second matching degree are combined to obtain a target matching degree between the form item and the trigger text;

[0025] The candidate form item corresponding to the trigger text is determined based on the target matching degree of the form item.

[0026] Optionally, the trigger type includes a passive type, and if a trigger operation for a virtual object is detected, determining the trigger type of the trigger operation, and displaying the virtual object while displaying the form to be filled in, comprises:

[0027] Based on the mouse event, determining an indication form item from the plurality of form items;

[0028] Detecting a length change trend of the content in the indication form item;

[0029] If the length variation trend conforms to a preset trend, it is determined that a passive type trigger operation is detected;

[0030] If the length change trend does not conform to the preset trend, calculating the maintenance time of the indication form item;

[0031] If the maintenance time is longer than the preset time, it is determined that an active type of trigger operation is detected;

[0032] If the passive type of trigger operation is detected, the virtual object and the form to be filled in are displayed in a second arrangement.

[0033] Optionally, the determining the indicating form item from the multiple form items based on the mouse event includes:

[0034] If a mouse event is detected, determining the event type corresponding to the mouse event;

[0035] If the event type is a specified type, the current position of the mouse and the element position corresponding to each form item are obtained, where the element position includes the element area position and the element center position;

[0036] If the current position overlaps with any element region position, determining the form item corresponding to the element region position as the indication form item;

[0037] If the current position does not overlap with the positions of all element regions, calculate the straight-line distance between the current position and the center position of each element to obtain multiple candidate distances;

[0038] Based on a minimum value among the plurality of candidate distances, an indicated form item is determined from the plurality of form items.

[0039] Optionally, the feedback information includes confirmation information, and the step of responding to the feedback information for the query information and controlling the virtual object to perform an auxiliary filling action for the target form item by using the feedback information includes:

[0040] If the feedback information includes confirmation information, obtaining target description information corresponding to the target form item;

[0041] Performing intent analysis on the feedback information to obtain an intent type corresponding to the feedback information;

[0042] If the intention type does not include the specified type, the query information, the feedback information and the target description information are integrated to generate and output auxiliary filling information;

[0043] If the intent type includes a specified type, multiple rounds of dialogue are triggered to obtain filled-in content, and the filled-in content is input into the target form item.

[0044] Optionally, the fusing the query information, the feedback information and the target description information to generate and output auxiliary filling information includes:

[0045] Using a first language model to polish the query information and the feedback information to generate a target question;

[0046] Using the second language model to infer the target question within the scope of the target description information to obtain auxiliary filling information;

[0047] The virtual object is controlled to output the auxiliary filling information.

[0048] Optionally, triggering multiple rounds of dialogue to obtain filled-in content and inputting the filled-in content into the target form item includes:

[0049] Based on the feedback information and the target description information corresponding to the target form item, generate a plurality of execution steps with a preset order;

[0050] Determining a current execution step from the plurality of execution steps according to the preset order;

[0051] Generate a dialogue content corresponding to the currently executed step, and control the virtual object to output the dialogue content;

[0052] If the feedback content of the dialogue content includes a specified response, returning to execute the step of determining the current execution step and subsequent steps from the multiple execution steps in the preset order until the specified responses corresponding to all the execution steps are obtained;

[0053] The specified response corresponding to the last execution step in the preset sequence is input into the target form item as the filled content.

[0054] On the other hand, an embodiment of the present invention further provides a form filling device based on a virtual object, which is used to implement any of the above methods, and the device includes:

[0055] A form display module, used to display a form to be filled in, wherein the form to be filled in includes a plurality of form items to be filled in;

[0056] An object display module, configured to, if a trigger operation for a virtual object is detected, determine a trigger type of the trigger operation, and display the virtual object while displaying the form to be filled in;

[0057] A target determination module, configured to determine a target form item from the plurality of form items based on a trigger type of the trigger operation, wherein the target form item is a form item that currently needs to be assisted in filling out;

[0058] An inquiry module, used to generate inquiry information by using a specified template and the target form item and control the virtual object to output the inquiry information;

[0059] A filling module is used to respond to feedback information for the query information and use the feedback information to control the virtual object to perform an auxiliary filling action for the target form item, wherein the auxiliary filling action includes outputting auxiliary filling information or filling in the target form item.

[0060] On the other hand, an embodiment of the present invention further provides an electronic device, including a processor and a memory, wherein the memory stores multiple instructions; the processor loads instructions from the memory to execute the steps in any virtual object-based form filling method provided in an embodiment of the present invention.

[0061] On the other hand, an embodiment of the present invention further provides a computer-readable storage medium, which stores multiple instructions, and the instructions are suitable for a processor to load to execute the steps in any virtual object-based form filling method provided by an embodiment of the present invention.

[0062] On the other hand, an embodiment of the present invention further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of any virtual object-based form filling method provided in an embodiment of the present invention.

[0063] The beneficial effects brought about by the technical solution provided by the present invention include at least:

[0064] The form filling method and device based on virtual objects provided by the embodiment of the present invention can display multiple form items in the form to be filled in. When a trigger operation for the virtual object is detected, the trigger type is determined and the virtual object is displayed. The target form item can be accurately determined by the trigger type, and the target form item and the specified template are combined to generate the query information of the virtual object, and then the virtual object is controlled to output the query information and interact with the user. In response to the user's feedback information, the virtual object is controlled to perform auxiliary filling actions, so as to timely assist the user in filling in the form during the user's filling in the form, which is convenient and fast, and can effectively improve the efficiency and accuracy of data collection. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0066] Figure 1 Schematic diagram of an application scenario of a form filling method based on a virtual object provided in an embodiment of the present invention;

[0067] Figure 2 is a flowchart of a form filling method based on a virtual object provided by an embodiment of the present invention;

[0068] Figure 3 It is a schematic diagram of a process of determining a target form item based on a trigger operation provided by an embodiment of the present invention;

[0069] Figure 4 is a schematic diagram of displaying a virtual object and a form to be filled in provided by an embodiment of the present invention;

[0070] Figure 5 is a schematic diagram of determining an indication form item provided by an embodiment of the present invention;

[0071] Figure 6 is a structural diagram of a form filling device based on a virtual object provided in an embodiment of the present invention;

[0072] Figure 7 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0073] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0074] The embodiment of the present invention proposes a form filling method based on virtual objects, which can provide timely assistance to users in the process of filling in the form, so as to collect data quickly and accurately.

[0075] It will be understood that in the specific implementation of the present invention, data related to user information, such as user operations, user voice, etc., require user permission or consent, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0076] See also Figure 1 , shows a schematic diagram of an application scenario of a form filling method based on a virtual object. The application scenario may include a terminal 101 and a server 102, and the terminal 101 and the server 102 may exchange data through a network. A corresponding platform may be run on the terminal 101, and the platform may provide a form to be filled in. The terminal 101 may be a mobile phone, a tablet computer, a smart Bluetooth device, a computer, a large screen device, a robot, etc.; the server 102 may be a single server or a server cluster consisting of multiple servers.

[0077] The form to be filled in can be displayed on the terminal 101, and the user can operate on the terminal 101 to fill in the form to be filled in. If a trigger operation for a virtual object is detected, the terminal 101 can forward the corresponding trigger operation to the server 102, so that the server 102 determines the operation type of the trigger operation and controls the terminal 101 to display the form to be filled in and the virtual object at the same time. Then the server 102 can determine the target form item from multiple form items based on the trigger type of the trigger operation, and the target form item is the form item that currently needs to be assisted in filling in; generate query information using the specified template and the target form item and control the virtual object to output the query information; in response to feedback information for the query information, use the feedback information to control the virtual object to perform an assisted filling action for the target form item, and the assisted filling action includes outputting assisted filling information or filling in the target form item.

[0078] In this embodiment, a form filling method based on virtual objects is provided, such as Figure 2 As shown, the specific process of the form filling method based on virtual objects can be as follows:

[0079] S110: Display a form to be filled in, wherein the form to be filled in includes a plurality of form items to be filled in.

[0080] A form to be filled in is a form that needs to be filled in by the user. The form to be filled in may include multiple form items to be filled in. Among them, the form can be used in a variety of scenarios, such as questionnaires, task configuration tables, various parameter configuration forms in model training, etc.

[0081] The form to be filled in can be provided by the platform and displayed in the graphical user interface provided by the terminal device. For example, the model training platform can provide users with training model services, and users can use the model training platform to display the form to be filled in provided by the model training platform.

[0082] S120: If a trigger operation for a virtual object is detected, determine a trigger type of the trigger operation, and display the virtual object while displaying the form to be filled in.

[0083] It is understandable that the form to be filled in contains multiple form items to be filled in, and different form items require users to fill in different required content. Users may not know the meaning and function of each form item and may feel confused when filling in the form items. Therefore, when the form to be filled in is displayed, the user's operation can be detected in real time to determine whether to display the virtual object.

[0084] A virtual object is an intelligent entity that uses a digital image, which can be a human image, a cartoon image, or an animal image, etc., and can be set according to actual needs. The virtual object can have interactive functions such as voice output and text output, and can assist users in filling out forms, for example, interacting with users to answer users' related questions, helping users fill out forms, etc.

[0085] The trigger operation can be set in advance according to actual needs, and there can be multiple trigger operations, that is, detecting any trigger operation can activate the virtual object. Different trigger operations are set with corresponding trigger types. In the embodiment of the present invention, the trigger type can include active type and passive type.

[0086] Among them, active type and passive type are divided from the user's perspective. Active type means that the user actively activates the virtual object by voice or clicking on the control. For example, the virtual object can be set with a corresponding wake-up word, and the user can activate the virtual object based on the wake-up word. For another example, an object activation control can be displayed while displaying a form to be filled in, and the user can click the object activation control to activate the virtual object. The passive type refers to detecting mouse events without the user's awareness, and automatically activating the virtual object when the mouse event meets certain conditions. Different trigger types have different detection methods for trigger operations, and the subsequent methods for determining the target form items are also different. Please refer to Figure 3 , shows a schematic diagram of the process of determining the target form item based on the trigger operation. Figure 3 Provide detailed explanation.

[0087] If a trigger operation for a virtual object is detected, the virtual object may be displayed, wherein the virtual object should be displayed together with the form to be filled in. Optionally, the virtual object and the form to be filled in may be displayed in a superimposed manner or side by side, wherein superimposed display means that there is an overlapping area between the virtual object and the form to be filled in, and side by side display means that there is no overlapping area between the virtual object and the form to be filled in.

[0088] As an implementation mode, the trigger type includes an active type. When detecting the trigger operation to determine the trigger type and display the virtual object, if a trigger voice is detected, the trigger voice is analyzed and processed to obtain the trigger intention; if the trigger intention is an intention to help, it is determined that an active type of trigger operation is detected; if the active type of trigger operation is detected, the virtual object and the form to be filled in are displayed in a first arrangement.

[0089] When a form to be filled out is displayed, the user's voice can be detected in real time, and the detected user voice can be used as a trigger voice. In order to improve the accuracy of activating virtual objects and avoid erroneous activation of virtual objects, the trigger voice can be analyzed and processed accordingly to determine whether an active type of trigger operation is detected.

[0090] Optionally, the triggering speech may be converted into text by speech recognition technology to obtain the triggering text, and then the triggering text may be subjected to intent analysis to obtain the triggering intent. A first intent analysis model may be pre-trained, and the first intent analysis model may output a first identifier when the triggering text is an intent to help, and output a second identifier when the triggering text is not an intent to help.

[0091] The trigger text can be input into the first intent analysis model. If the first intent analysis model outputs the first identifier, it can be determined that an active type of trigger operation is detected; if the first intent analysis model outputs the second identifier, no processing is performed and new trigger voices continue to be detected.

[0092] If an active type of trigger operation is detected, a virtual object can be called out from the page currently displaying the form to be filled in, so as to display the form to be filled in and the virtual object together. The virtual object and the form to be filled in can be displayed in a first arrangement, and the first arrangement can be to display the form to be filled in on the left side of the virtual object, to display the form to be filled in on the upper side of the virtual object, etc., which can be set according to actual needs and is not specifically limited here. For example, see Figure 4 , shows a schematic diagram of displaying a virtual object and a form to be filled in, wherein the virtual object is displayed on the left side of the form to be filled in.

[0093] In some embodiments, the trigger type may also include a passive type. When detecting a trigger operation to determine the trigger type and displaying a virtual object, it may be possible to determine an indication form item from the multiple form items based on a mouse event; detect a length change trend of the content in the indication form item; if the length change trend conforms to a preset trend, determine that a passive type of trigger operation is detected; if the length change trend does not conform to the preset trend, calculate the maintenance time of the indication form item; if the maintenance time is greater than the preset time, determine that an active type of trigger operation is detected; if the passive type of trigger operation is detected, display the virtual object and the form to be filled in according to a second arrangement.

[0094] The form to be filled out may include multiple form items, and each form item is displayed on the page through corresponding elements, which can be text boxes, drop-down boxes, etc. Mouse events refer to events related to the mouse, which are used to respond to user operations. Corresponding mouse events will be generated when the user moves the mouse or clicks the mouse.

[0095] Based on the mouse event, an indicating form item may be determined from a plurality of form items, where the indicating form item is a form item indicated by a mouse cursor obtained after analyzing the mouse event.

[0096] As an implementation manner, when determining an indication form item based on a mouse event, if a mouse event is detected, determining an event type corresponding to the mouse event; if the event type is a specified type, obtaining a current position of the mouse and an element position corresponding to each form item, wherein the element position includes an element area position and an element center position; if the current position overlaps with any element area position, determining the form item corresponding to the element area position as an indication form item; if the current position does not overlap with any element area positions, calculating a straight-line distance between the current position and each element center position to obtain multiple candidate distances; and determining an indication form item from the multiple form items based on a minimum value among the multiple candidate distances.

[0097] Different mouse operations generate different event types. In the embodiment of the present invention, the event types may include: click type, mouse press type, mouse move-in type, mouse move-out type, etc. When a mouse event is detected, the event type corresponding to the mouse event can be obtained. The specified type is a pre-set type of mouse event that needs to be paid attention to. It can be set according to actual needs and is not specifically limited here. For example, in the embodiment of the present invention, the specified type may be a combination of click type, mouse move-in type and mouse move-out type.

[0098] If the event type is the specified type, it can be considered that a mouse event that needs attention is detected to obtain the current position of the mouse cursor. For each form item in the form to be filled, the document object model of the form to be filled can be obtained, and through this document object model, the hierarchical information, position information, etc. corresponding to each element in the form to be filled can be obtained.

[0099] Among them, the element position and the current position are position coordinates in the same coordinate system. The element position can include the element area position and the element center position. The element area position can include the upper left coordinate and the lower right coordinate of the element, and the element center position is the coordinate corresponding to the center of the element, where the element center position can be calculated based on the element area position.

[0100] For each form item, the current position can be compared with the element area position of the form item to determine whether the current position is within the area enclosed by the upper left coordinate and the lower right coordinate. If it is within this area, the current position and the element area position overlap, indicating that the mouse cursor points to this element, and then the form item corresponding to the element area position can be directly used as the indicated form item. For example, reference can be made to Figure 5 , which shows a schematic diagram for determining the indicated form item. Figure 5 In, 3 form items are exemplified. The 3 squares are the element area positions corresponding to the 3 form items respectively. Among them, A1 is the element center position of form item 1, A2 is the element center position of form item 2, and A3 is the element center position of form item 3. If the current cursor position is O1, obviously O1 overlaps with the element area position of form item 1, and form item 1 can be directly determined as the indicated form item.

[0101] If the current position is not within any element area position, that is, the current position does not overlap with all element area positions, that is, the mouse cursor may not point to a form item, the form item closest to the cursor can be used as the indicated form item. As an implementation method, the straight-line distance between the current position and each element center position can be calculated to determine the form item closest to the cursor. For example, by calculating the straight-line distance between the current position and each element center position, multiple candidate distances can be obtained; then the smallest candidate distance can be determined from the multiple candidate distances, and the element center position corresponding to the smallest candidate distance can be obtained, and then the form item corresponding to the element center position can be determined, and this form item is the indicated form item. Continue to refer to Figure 5 , if the current cursor position is O2, the straight-line distances d1, d2, and d3 can be obtained. Obviously, d3 < d2 < d1, so form item 3 can be determined as the indicated form item.

[0102] It should be noted that the indicated form item detected in the above manner may be a form item on which the mouse is hovering, a form item on which the mouse is clicking, or a form item in which content is being input. If the user is filling in content in the indicated form item, the length of the content in the indicated form item will change, and the length change trend of the content in the indicated form item may be detected first.

[0103] The preset trend refers to the trend that the length of the content in the form item may show when the user has difficulty filling out the form correctly. The length of the content can be measured by the number of characters. When the user has difficulty filling out the form correctly, he or she may repeatedly delete the content and fill it out again, and the length of the content will show a trend of getting longer, shorter, longer, and shorter. For example, the user enters 5 characters, then deletes 2 characters and leaves 3 characters. The trend is shorter based on 5 characters; the user continues to enter 7 characters based on 3 characters, for a total of 10 characters. The trend is longer based on 3 characters; the user deletes 4 characters again, leaving 6 characters. The trend is shorter based on 10 characters.

[0104] Thus, the length change trend of the content in the indication form item can be detected to determine the length change trend. If the length change trend meets the preset trend, it is determined that an active type of trigger operation is detected. If the length change trend does not meet the preset trend, the maintenance time of the indication form item can be calculated. The maintenance time of the indication form item refers to the time when the form item is confirmed as the indication form item, that is, the time when the indication form item is determined can be obtained, and the maintenance time can be obtained by subtracting the time from the current time.

[0105] If the duration is longer than the preset duration, it means that the cursor has been hovering over the indicated form item, or the indicated form item has been selected but no other operations have been performed. The user may be thinking about how to fill in the form. At this time, it can be determined that a passive type of trigger operation has been detected. Among them, the preset duration can be set according to actual needs, or obtained through a large number of experimental statistics, and no specific limitation is made here. If a passive type of trigger operation is detected, the virtual object form to be filled in can be displayed in a second arrangement mode, wherein the second arrangement mode can be the same as the first arrangement mode, or it can be different, and can be set according to actual needs, and no specific limitation is made here.

[0106] If the holding time is not greater than the preset time and the mouse event changes, the new mouse event can be detected for judgment; if the mouse event still does not change, the holding time is continued to be calculated after waiting for a period of time.

[0107] The above-mentioned active type and passive type trigger operations can be executed selectively, among which the active type has a higher priority than the passive type. That is, if an active type trigger operation is detected during the process of detecting a passive type trigger operation, the subsequent process will be executed with the active type trigger operation.

[0108] S130: Determine a target form item from the multiple form items based on the trigger type of the trigger operation.

[0109] The target form item refers to the form item that currently needs to be filled in with assistance. The target form item is related to the user operation. The trigger type of the aforementioned trigger operation can include active type and passive type. Different trigger types have different ways of determining the target form item. Please continue to refer to Figure 3 As an implementation manner, if the trigger type is an active type, determining the target form item may be to convert the trigger voice into a trigger text; based on the trigger text and the description information corresponding to each of the form items, detecting a candidate form item corresponding to the trigger text from the multiple form items; if the candidate form item is detected, determining the candidate form item as the target form item; if the candidate form item is not detected, determining an indicated form item from the multiple form items based on a mouse event, and using the indicated form item as the target form item, wherein the indicated form item is the form item indicated by the cursor.

[0110] When the above-mentioned determination detects an active type of trigger operation, the trigger voice will be analyzed for intent. At this time, the trigger voice can also be converted into trigger text and analyzed to determine the target form item. Among them, each form item is pre-set with its corresponding description information, which may include a specific description of the form item and an explanation for filling in the form item. Among them, the specific description may refer to the description of the content that needs to be filled in the form item, and the explanation for filling in may refer to the relevant explanation of the content that needs to be filled in the form item. For example, for the mailbox form item, the specific description may be "Enter your email address", and the explanation for filling in may be "The email address is your private mailbox for receiving important notifications."

[0111] Based on the trigger text and the description information of each form item, the target matching degree of each form item and the trigger text can be calculated, so as to determine the candidate form items based on the target matching degree. For example, for each form item, the first matching degree between the specific description and the trigger text is calculated; the second matching degree between the filling explanation and the trigger text is calculated; the first matching degree and the second matching degree are combined to obtain the target matching degree between the form item and the trigger text; and the candidate form item corresponding to the trigger text is determined based on the target matching degree of the form item.

[0112] The specific description, explanation, and trigger text are all in text form. The text vectorization model can be used to convert the three into corresponding vector representations, and then the cosine similarity between the specific description vector and the trigger text vector is calculated as the first matching degree. The larger the cosine similarity, the higher the first matching degree, that is, the more matching the specific description and the trigger text. Similarly, the cosine similarity between the explanation vector and the trigger text vector is calculated as the second matching degree. The larger the cosine similarity, the higher the second matching degree, that is, the more matching the explanation and the trigger text.

[0113] The target matching degree can be obtained by fusing the first matching degree and the second matching degree. As an implementation method, corresponding weights can be set in advance for the specific description and the filling explanation according to actual needs. For example, the weight corresponding to the specific description is the first weight, and the weight corresponding to the filling explanation is the second weight. The first candidate matching degree is obtained by multiplying the first weight and the first matching degree, and the second candidate matching degree is obtained by multiplying the second weight and the second matching degree. The first candidate matching degree and the second candidate matching degree are then summed to obtain the target matching degree.

[0114] Each form item and trigger text can be calculated to obtain a corresponding target matching degree, and the target matching degree of each form item can be used to determine the candidate form item. Based on the meaning of the first matching degree and the second matching degree, it can be seen that the higher the target matching degree, the more the description information of the trigger text and the form item matches, indicating that the user's trigger voice and the corresponding form item are more relevant. Therefore, the maximum target matching degree can be determined from multiple target matching degrees, and the form item corresponding to the maximum target matching degree can be determined as the candidate form item.

[0115] If a candidate form item is detected, it means that the trigger voice clearly contains content related to the form item, and the candidate form item can be directly determined as the target form item. For example, for the form item "email", the trigger voice may be what format the email is in.

[0116] If no candidate form item is detected, it indicates that the trigger voice may not explicitly mention the content related to the form item. For example, the user's trigger voice is "How do I fill this in" or "How do I write it?". This type of trigger voice does not contain the relevant content of a form item, so the candidate form item cannot be detected from the trigger voice. At this time, the indication form item can be determined from multiple form items based on the mouse event. The implementation of this part is consistent with the above. For details, please refer to the corresponding description above, which will not be repeated here. The indication form item is obtained based on the mouse event, indicating that the current cursor is at the indication form item, and the indication form item can be used as the target form item.

[0117] If the trigger type is a passive type, the corresponding indication form item may be directly determined as the target form item.

[0118] S140: Generate query information using a specified template and the target form item and control the virtual object to output the query information.

[0119] The designated template is a template of a preset inquiry message, which may include a form slot. For example, the designated template may be "Hello, do you want to fill in the [form slot] part?". The name of the target form item may be filled in the form slot to generate the inquiry message. For example, if the name of the target form item is an email address, the inquiry message is "Hello, do you want to fill in the email address part?".

[0120] Then, the virtual object can be controlled to output the inquiry information, wherein the virtual object can output the inquiry information in the form of text output, voice output, etc., which can be set according to actual needs. In the embodiment of the present invention, only voice output is used as an example for explanation. That is, the virtual object can broadcast the inquiry information by voice so as to interact with the user.

[0121] S150: In response to feedback information for the query information, use the feedback information to control the virtual object to perform an auxiliary filling action for the target form item.

[0122] After the virtual object outputs the inquiry information, the user will usually give corresponding feedback information, which may include confirmation information or denial information of the inquiry information. Different feedback information may have different corresponding subsequent processing. For example, if the feedback information includes denial information, indicating that the target form item is incorrectly confirmed, the virtual object may clearly ask the user for the target form item to be filled in, and after receiving the target form item feedback from the user, the auxiliary filling action for the target form item is performed.

[0123] If the feedback information contains confirmation information, it indicates that the target form item is confirmed correctly, and the auxiliary filling action for the target form item can be directly executed. Among them, the auxiliary filling action can include outputting auxiliary filling information or filling in the target form item. Outputting auxiliary filling information means generating and outputting explanation information for the user to fill in the target form item so that the user can better understand the form item; filling in the target form item means automatically filling the corresponding content into the target form item without the user filling in.

[0124] As an implementation method, when feedback information is used to control a virtual object to perform an auxiliary filling action on a target form item, if the feedback information contains confirmation information, the target description information corresponding to the target form item is obtained; the feedback information is subjected to intent analysis to obtain the intent type corresponding to the feedback information; if the intent type does not contain a specified type, the confirmation information and the target description information are integrated to generate auxiliary filling information; if the intent type contains a specified type, multiple rounds of dialogue are triggered to obtain the filled-in content, and the filled-in content is input into the target form item.

[0125] In response to the confirmation information for the inquiry information, the description information corresponding to the target form item can be obtained. As mentioned above, the description information can include a specific description and filling explanation. The description information can also include a unique identifier of the form item, an association relationship, etc. Among them, the association relationship refers to the relationship between the form item and other form items, which may be a mutually exclusive relationship or a dependent relationship, etc. For the convenience of description, the description information corresponding to the target form item can be called the target description information.

[0126] When the feedback information contains confirmation information, the feedback information can continue to be analyzed for intent. For example, if the user inputs feedback information through voice, the feedback voice can be converted into feedback text, and then the pre-trained intent recognition model can be used to analyze the intent type corresponding to the feedback information. The intent recognition model can be a large language model, which can be pre-set with multiple intent types and trained using a large amount of sample data.

[0127] The designated type is a preset intention type. In the embodiment of the present invention, the designated type may be an instruction type, that is, the feedback information contains corresponding filling instruction information. For example, the user's feedback information is "Yes, please help me fill it out", which contains a clear instruction to help fill it out.

[0128] If the intent type corresponding to the feedback information does not include the specified type, it means that the user's feedback information is only to confirm the target form item, and it is necessary to obtain help for filling in the target form item. For example, the user's feedback information is "Yes, what does this mean? How do I fill it in?".

[0129] In order to ensure accurate answers to user questions, the inquiry information, feedback information and target description information can be integrated to generate and output auxiliary filling information. As an implementation method, the inquiry information and the feedback information can be polished using a first language model to generate a target question; the target question can be inferred using a second language model within the scope of the target description information to obtain auxiliary filling information; and the virtual object can be controlled to output the auxiliary filling information.

[0130] The first language model and the second language model can be different language models for implementing different functions. The first language model is used to polish the text, and can fuse the inquiry information and the feedback information to form a complete and comprehensive text description, that is, the target question. This method can avoid missing the user's needs when the feedback information contains follow-up questions.

[0131] The generated target question can be input into the second language model, which is used to generate an answer that matches the target question based on specific knowledge to obtain auxiliary filling information. This method can control the second language model to answer the target question based on the target description information, avoid answering irrelevant questions, effectively reduce model hallucinations, and improve the accuracy of generating auxiliary filling information. The auxiliary filling information can then be broadcasted through a virtual object so that the user can obtain the corresponding information.

[0132] If the intent type includes the specified type, it indicates that the user needs the virtual object to fill in the target form item. In order to ensure the accuracy of the filled-in content, multiple rounds of dialogue can be triggered. Through multiple rounds of dialogue with the user, the content that needs to be filled in can be obtained, and then the corresponding content is filled in the target form item.

[0133] As an implementation method, when multiple rounds of dialogues are triggered to obtain filled-in content and the filled-in content is input into a target form item, multiple execution steps with a preset order can be generated based on the feedback information and the target description information corresponding to the target form item; the current execution step is determined from the multiple execution steps according to the preset order; dialogue content corresponding to the current execution step is generated, and the virtual object is controlled to output the dialogue content; if the feedback content of the dialogue content contains a specified response, return to execute the step of determining the current execution step from the multiple execution steps according to the preset order and subsequent steps until the specified responses corresponding to all execution steps are obtained; and the response content corresponding to the last execution step in the preset order is input into the target form item as the filled-in content.

[0134] By integrating the feedback information and the target description information corresponding to the target form item, the user's needs can be accurately obtained. This step can be processed by the preset large language model. The feedback information and the target description information are input into the large language model together. The large language model can analyze the user's needs and disassemble the user's needs so as to gradually realize the user's needs. The disassembled user needs are multiple execution steps with a preset order. For example, the user's need is for the virtual object to help fill in the form item of the email address. The execution steps can be to first explain the meaning of the form item and ask the user if he has an email address; if there is no email address, guide the user to register; if there is an email address, obtain the email address through a dialogue; confirm the email address, and fill it in after confirmation. The execution steps obtained by disassembling the above process are: explain the meaning, ask for the email address.

[0135] Specifically, the execution steps can be determined as the current execution steps in sequence according to the preset order. For example, in the above example, "explain the meaning" can be determined as the current execution step. The large language model can generate the corresponding dialogue content using the target description information, that is, the relevant explanation of "email" and ensure that the user has no questions about the meaning. If there are no questions, it can be set as the designated response for the "explain the meaning". When the user thinks there are no questions, "ask for email address" can be set as the current execution step to generate the dialogue content and user interaction of "ask for email address". The designated response of "ask for email address" can be to obtain the correct email address. When the large language model obtains the correct email address, the obtained email address can be filled into the target form item. At this time, the virtual object can be controlled to play the corresponding filling animation to give the user a visual effect that the virtual object fills the content into the target form item.

[0136] It should be noted that during the above-mentioned multiple rounds of conversations, the user's feedback content needs to be analyzed for intent type each time in order to determine whether the new feedback content contains the specified type of intent type, and then the auxiliary filling action can be adjusted in time to ensure that the auxiliary filling action matches the user's needs.

[0137] After controlling the virtual object to perform the auxiliary filling action, the user can actively fill in the form items or the virtual object can fill in the form items. Each form item will trigger the corresponding check after filling in to ensure the rationality of the content filled in the current form item. When the user submits the form to be filled in, the entire check can be performed, and when a form item is detected to be abnormal, the abnormal form item is automatically located.

[0138] The form filling scheme based on virtual objects provided by the embodiment of the present invention can be applied in various scenarios where data collection requires the use of forms. For example, taking the collection of relevant configuration parameters for model training in the model training platform as an example, since the meanings of most configuration parameters are relatively obscure, the scheme provided by the embodiment of the present invention can provide real-time assistance to users through virtual objects when users fill in the configuration parameters, without the need for users to ask others or use the Internet to search, which is more real-time and efficient, and the virtual object can generate corresponding auxiliary information based on the information corresponding to the form, which is more accurate and comprehensive, and is convenient for quickly filling in forms, thereby improving the efficiency and accuracy of data collection.

[0139] Through the method provided by the embodiment of the present invention, a virtual object can be introduced into the form, and the user's operation can be detected in real time, and the virtual object can be displayed in a timely manner, and the user can be assisted in filling out the form through the interaction between the virtual object and the user. The content of the interaction between the virtual object and the user can be generated based on the corresponding description information of the form, so that the interactive content is more in line with the design of the form and has higher accuracy. In addition, the functions of the virtual object are diverse, and it can not only communicate in real time but also help users fill out the form, which is more convenient and efficient, thereby improving the efficiency and accuracy of form filling, so as to collect data quickly and accurately.

[0140] In order to better implement the above method, the embodiment of the present invention also provides a form filling device based on a virtual object, which can be integrated in an electronic device, and the electronic device can be a terminal, a server, etc. Among them, the terminal can be a mobile phone, a tablet computer, a smart Bluetooth device, a laptop, a personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers.

[0141] For example, in this embodiment, the method of the embodiment of the present invention is described in detail by taking the virtual object-based form filling device specifically integrated in the server as an example.

[0142] For example, Figure 6 As shown, the virtual object-based form filling device 200 may include a form display module 210 , an object display module 220 , a target determination module 230 , an inquiry module 240 , and a filling module 250 .

[0143] The form display module 210 is used to display a form to be filled in, wherein the form to be filled in includes a plurality of form items to be filled in;

[0144] The object display module 220 is used to determine the trigger type of the trigger operation if a trigger operation for the virtual object is detected, and display the virtual object while displaying the form to be filled in;

[0145] A target determination module 230, configured to determine a target form item from the plurality of form items based on a trigger type of the trigger operation, wherein the target form item is a form item that currently needs to be assisted in filling out;

[0146] The query module 240 is used to generate query information using a specified template and the target form item and control the virtual object to output the query information;

[0147] The filling module 250 is used to respond to the feedback information for the query information and use the feedback information to control the virtual object to perform an auxiliary filling action for the target form item, wherein the auxiliary filling action includes outputting auxiliary filling information or filling in the target form item.

[0148] In some embodiments, the trigger type includes an active type, and the object display module 220 is specifically used to:

[0149] If a triggering voice is detected, the triggering voice is analyzed and processed to obtain a triggering intention;

[0150] If the triggering intention is an intention to seek help, determining that an active type of triggering operation is detected;

[0151] If the active type of trigger operation is detected, the virtual object and the form to be filled out are displayed in a first arrangement.

[0152] In some embodiments, the target determination module 230 is specifically configured to:

[0153] Converting the trigger voice into a trigger text;

[0154] Based on the trigger text and the description information corresponding to each of the form items, detecting a candidate form item corresponding to the trigger text from the multiple form items;

[0155] If the candidate form item is detected, determining the candidate form item as a target form item;

[0156] If the candidate form item is not detected, based on the mouse event, an indicated form item is determined from the multiple form items, and the indicated form item is used as the target form item, where the indicated form item is the form item indicated by the cursor.

[0157] In some embodiments, the description information includes a specific description and an explanation, and the target determination module 230 is specifically used to:

[0158] For each of the form items, calculating a first matching degree between the specific description and the trigger text;

[0159] Calculating a second matching degree between the filling explanation and the trigger text;

[0160] The first matching degree and the second matching degree are combined to obtain a target matching degree between the form item and the trigger text;

[0161] The candidate form item corresponding to the trigger text is determined based on the target matching degree of the form item.

[0162] In some embodiments, the trigger type includes a passive type, and the object display module 220 is specifically used to:

[0163] Based on the mouse event, determining an indication form item from the plurality of form items;

[0164] Detecting a length change trend of the content in the indication form item;

[0165] If the length variation trend conforms to a preset trend, it is determined that a passive type trigger operation is detected;

[0166] If the length change trend does not conform to the preset trend, calculating the maintenance time of the indication form item;

[0167] If the maintenance time is longer than the preset time, it is determined that an active type of trigger operation is detected;

[0168] If the passive type of trigger operation is detected, the virtual object and the form to be filled in are displayed in a second arrangement.

[0169] In some embodiments, the object display module 220 or the target determination module 230 is specifically configured to:

[0170] If a mouse event is detected, determining the event type corresponding to the mouse event;

[0171] If the event type is a specified type, the current position of the mouse and the element position corresponding to each form item are obtained, where the element position includes the element area position and the element center position;

[0172] If the current position overlaps with any element region position, determining the form item corresponding to the element region position as the indication form item;

[0173] If the current position does not overlap with the positions of all element regions, calculate the straight-line distance between the current position and the center position of each element to obtain multiple candidate distances;

[0174] Based on a minimum value among the plurality of candidate distances, an indicated form item is determined from the plurality of form items.

[0175] In some embodiments, the feedback information includes confirmation information, and the filling module 250 is specifically used to:

[0176] If the feedback information includes confirmation information, obtaining target description information corresponding to the target form item;

[0177] Performing intent analysis on the feedback information to obtain an intent type corresponding to the feedback information;

[0178] If the intention type does not include the specified type, the query information, the feedback information and the target description information are integrated to generate and output auxiliary filling information;

[0179] If the intent type includes a specified type, multiple rounds of dialogue are triggered to obtain filled-in content, and the filled-in content is input into the target form item.

[0180] In some embodiments, the filling module 250 is specifically used to:

[0181] Using a first language model to polish the query information and the feedback information to generate a target question;

[0182] Using the second language model to infer the target question within the scope of the target description information to obtain auxiliary filling information;

[0183] The virtual object is controlled to output the auxiliary filling information.

[0184] In some embodiments, the filling module 250 is specifically used to:

[0185] Based on the feedback information and the target description information corresponding to the target form item, generate a plurality of execution steps with a preset order;

[0186] Determining a current execution step from the plurality of execution steps according to the preset order;

[0187] Generate a dialogue content corresponding to the currently executed step, and control the virtual object to output the dialogue content;

[0188] If the feedback content of the dialogue content includes a specified response, returning to execute the step of determining the current execution step and subsequent steps from the multiple execution steps in the preset order until the specified responses corresponding to all the execution steps are obtained;

[0189] The specified response corresponding to the last execution step in the preset sequence is input into the target form item as the filled content.

[0190] In specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined and implemented as the same or several entities. The specific implementation of the above modules can be found in the previous method embodiments, which will not be repeated here.

[0191] As can be seen from the above, the form filling device based on virtual objects in this embodiment can display multiple form items in the form to be filled in. When a trigger operation for a virtual object is detected, the trigger type is determined and the virtual object is displayed. The trigger type can be used to accurately determine the target form item, and the target form item and the specified template are combined to generate the query information of the virtual object, and then the virtual object is controlled to output the query information and interact with the user. In response to the user's feedback information, the virtual object is controlled to perform auxiliary filling actions, so as to timely assist the user in filling in the form during the user's filling in the form, which is convenient and fast, and can effectively improve the efficiency and accuracy of data collection.

[0192] The embodiment of the present invention further provides an electronic device, which may be a terminal, a server, or the like. The terminal may be a mobile phone, a tablet computer, a smart Bluetooth device, a notebook computer, a personal computer, or the like; the server may be a single server or a server cluster composed of multiple servers, or the like.

[0193] In some embodiments, the virtual object-based form filling device can also be integrated into multiple electronic devices. For example, the virtual object-based form filling device can be integrated into multiple servers, and the virtual object-based form filling method of the present invention can be implemented by multiple servers.

[0194] In this embodiment, the electronic device of this embodiment is a server as an example for detailed description, for example, Figure 7 As shown, it shows a schematic diagram of the structure of an electronic device involved in an embodiment of the present invention, specifically:

[0195] The electronic device may include components such as a processor 310 with one or more processing cores, a memory 320 with one or more computer-readable storage media, a power supply 330, an input module 340, and a communication module 350. Those skilled in the art will appreciate that Figure 7 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0196] The processor 310 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 320, and calling data stored in the memory 320. In some embodiments, the processor 310 may include one or more processing cores; in some embodiments, the processor 310 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 310.

[0197] The memory 320 can be used to store software programs and modules. The processor 310 executes various functional applications and data processing by running the software programs and modules stored in the memory 320. The memory 320 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 320 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 320 may also include a memory controller to provide the processor 310 with access to the memory 320.

[0198] The electronic device also includes a power supply 330 for supplying power to various components. In some embodiments, the power supply 330 can be logically connected to the processor 310 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 330 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0199] The electronic device may further include an input module 340, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0200] The electronic device may further include a communication module 350. In some embodiments, the communication module 350 may include a wireless module. The electronic device may perform short-range wireless transmission through the wireless module of the communication module 350, thereby providing the user with wireless broadband Internet access. For example, the communication module 350 may be used to help the user send and receive emails, browse web pages, and access streaming media.

[0201] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 310 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 320 according to the following instructions, and the processor 310 will run the application programs stored in the memory 320, thereby implementing the steps in the methods of the embodiments of the present invention.

[0202] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0203] As can be seen from the above, the electronic device of this embodiment can display multiple form items in the form to be filled in. When a trigger operation for a virtual object is detected, the trigger type is determined and the virtual object is displayed. The target form item can be accurately determined by the trigger type, and the target form item and the specified template are combined to generate the query information of the virtual object, and then the virtual object is controlled to output the query information and interact with the user. In response to the user's feedback information, the virtual object is controlled to perform the auxiliary filling action, so as to timely assist the user to fill in the form during the user's filling in the form, which is convenient and fast, and can effectively improve the efficiency and accuracy of data collection.

[0204] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0205] To this end, an embodiment of the present invention provides a computer-readable storage medium, in which a plurality of instructions are stored. The instructions can be loaded by a processor to execute the steps in any virtual object-based form filling method provided by an embodiment of the present invention.

[0206] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0207] According to one aspect of the present invention, a computer program product or computer program is provided, the computer program product or computer program including a computer program / instruction, the computer program / instruction being stored in a computer-readable storage medium. A processor of an electronic device reads the computer program / instruction from the computer-readable storage medium, and the processor executes the computer program / instruction, so that the electronic device executes the method provided in various optional implementations of the form filling based on virtual objects provided in the above embodiments.

[0208] Since the instructions stored in the storage medium can execute the steps in any virtual object-based form filling method provided in the embodiments of the present invention, the beneficial effects that can be achieved by any virtual object-based form filling method provided in the embodiments of the present invention can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0209] The above is a detailed introduction to a form filling method and device based on virtual objects provided in an embodiment of the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A form filling method based on virtual objects, characterized in that: The method comprises: Displaying a form to be filled in, wherein the form to be filled in includes a plurality of form items to be filled in; If a trigger operation for a virtual object is detected, determining a trigger type of the trigger operation, and displaying the virtual object while displaying the form to be filled in, the trigger type includes an active type, and the active type of trigger operation is obtained based on trigger voice detection; Based on the trigger type of the trigger operation, determining a target form item from the multiple form items, the target form item being the form item that currently needs to be assisted in filling out; Filling the name of the target form item into a form slot of a specified template to generate query information, and controlling the virtual object to output the query information, wherein the specified template is a template of the query information set in advance; In response to feedback information for the inquiry information, using the feedback information to control the virtual object to perform an auxiliary filling action for the target form item, the auxiliary filling action includes outputting auxiliary filling information or filling the target form item, and the feedback information includes confirmation information; Wherein, determining the target form item from the multiple form items based on the trigger type of the trigger operation includes: converting the trigger voice into a trigger text; based on the trigger text and description information corresponding to each of the form items, detecting a candidate form item corresponding to the trigger text from the multiple form items, wherein the description information includes a specific description and a filling explanation; if the candidate form item is detected, determining the candidate form item as the target form item; if the candidate form item is not detected, determining an indicated form item from the multiple form items based on a mouse event, and using the indicated form item as the target form item, wherein the indicated form item is the form item indicated by the cursor; The detecting of candidate form items corresponding to the trigger text from the multiple form items based on the trigger text and the description information corresponding to each of the form items includes: for each of the form items, calculating a first matching degree between the specific description and the trigger text; calculating a second matching degree between the filling explanation and the trigger text; fusing the first matching degree and the second matching degree to obtain a target matching degree between the form item and the trigger text; and determining the candidate form items corresponding to the trigger text based on the target matching degree of the form item; The response to the feedback information for the inquiry information is to use the feedback information to control the virtual object to perform an auxiliary filling action for the target form item, including: if the feedback information contains confirmation information, obtaining the target description information corresponding to the target form item; performing intent analysis on the feedback information to obtain the intent type corresponding to the feedback information; if the intent type does not contain a specified type, fusing the inquiry information, the feedback information and the target description information to generate and output auxiliary filling information; if the intent type contains a specified type, triggering multiple rounds of dialogue to obtain filling content, and inputting the filling content into the target form item.

2. The method according to claim 1, characterized in that: The trigger type includes an active type, and if a trigger operation for a virtual object is detected, determining the trigger type of the trigger operation, and displaying the virtual object while displaying the form to be filled in, includes: If a triggering voice is detected, the triggering voice is analyzed and processed to obtain a triggering intention; If the triggering intention is an intention to seek help, determining that an active type of triggering operation is detected; If the active type of trigger operation is detected, the virtual object and the form to be filled out are displayed in a first arrangement.

3. The method according to claim 1, characterized in that The trigger type includes a passive type, and if a trigger operation for a virtual object is detected, determining the trigger type of the trigger operation, and displaying the virtual object while displaying the form to be filled in, includes: Based on the mouse event, determining an indication form item from the plurality of form items; Detecting a length change trend of the content in the indication form item; If the length variation trend conforms to a preset trend, it is determined that a passive type trigger operation is detected; If the length change trend does not conform to the preset trend, calculating the maintenance time of the indication form item; If the maintenance time is longer than the preset time, it is determined that an active type of trigger operation is detected; If the passive type of trigger operation is detected, the virtual object and the form to be filled in are displayed in a second arrangement.

4. The method according to claim 1 or 3, characterized in that: The step of determining the indicating form item from the plurality of form items based on the mouse event comprises: If a mouse event is detected, determining the event type corresponding to the mouse event; If the event type is a specified type, the current position of the mouse and the element position corresponding to each form item are obtained, where the element position includes the element area position and the element center position; If the current position overlaps with any element region position, determining the form item corresponding to the element region position as the indication form item; If the current position does not overlap with the positions of all element regions, calculate the straight-line distance between the current position and the center position of each element to obtain multiple candidate distances; Based on a minimum value among the plurality of candidate distances, an indicated form item is determined from the plurality of form items.

5. The method according to claim 1, characterized in that The step of fusing the query information, the feedback information and the target description information to generate and output auxiliary filling information includes: Using a first language model to polish the query information and the feedback information to generate a target question; Using the second language model to infer the target question within the scope of the target description information to obtain auxiliary filling information; The virtual object is controlled to output the auxiliary filling information.

6. The method according to claim 5, characterized in that The triggering of multiple rounds of dialogue to obtain filled content and inputting the filled content into the target form item includes: Based on the feedback information and the target description information corresponding to the target form item, generate a plurality of execution steps with a preset order; Determining a current execution step from the plurality of execution steps according to the preset order; Generate a dialogue content corresponding to the currently executed step, and control the virtual object to output the dialogue content; If the feedback content of the dialogue content includes a specified response, returning to execute the step of determining the current execution step and subsequent steps from the multiple execution steps in the preset order until the specified responses corresponding to all the execution steps are obtained; The specified response corresponding to the last execution step in the preset sequence is input into the target form item as the filled content.

7. A form filling device based on virtual objects, the device being used to implement the method according to any one of claims 1 to 6, characterized in that: The device comprises: A form display module, used to display a form to be filled in, wherein the form to be filled in includes a plurality of form items to be filled in; An object display module is used to determine the trigger type of the trigger operation if a trigger operation for the virtual object is detected, and display the virtual object while displaying the form to be filled in, wherein the trigger type includes an active type, and the trigger operation of the active type is obtained based on the trigger voice detection; A target determination module, configured to determine a target form item from the plurality of form items based on a trigger type of the trigger operation, wherein the target form item is a form item that currently needs to be assisted in filling out; An inquiry module, used to fill the name of the target form item into the form slot of the specified template to generate inquiry information, and control the virtual object to output the inquiry information, wherein the specified template is a template of the preset inquiry information; a filling module, configured to respond to feedback information for the query information, and use the feedback information to control the virtual object to perform an auxiliary filling action for the target form item, wherein the auxiliary filling action includes outputting auxiliary filling information or filling the target form item, and the feedback information includes confirmation information; Wherein, determining the target form item from the multiple form items based on the trigger type of the trigger operation includes: converting the trigger voice into a trigger text; based on the trigger text and description information corresponding to each of the form items, detecting a candidate form item corresponding to the trigger text from the multiple form items, wherein the description information includes a specific description and a filling explanation; if the candidate form item is detected, determining the candidate form item as the target form item; if the candidate form item is not detected, determining an indicated form item from the multiple form items based on a mouse event, and using the indicated form item as the target form item, wherein the indicated form item is the form item indicated by the cursor; The detecting of candidate form items corresponding to the trigger text from the multiple form items based on the trigger text and the description information corresponding to each of the form items includes: for each of the form items, calculating a first matching degree between the specific description and the trigger text; calculating a second matching degree between the filling explanation and the trigger text; fusing the first matching degree and the second matching degree to obtain a target matching degree between the form item and the trigger text; and determining the candidate form items corresponding to the trigger text based on the target matching degree of the form item; The response to the feedback information for the inquiry information is to use the feedback information to control the virtual object to perform an auxiliary filling action for the target form item, including: if the feedback information contains confirmation information, obtaining the target description information corresponding to the target form item; performing intent analysis on the feedback information to obtain the intent type corresponding to the feedback information; if the intent type does not contain a specified type, fusing the inquiry information, the feedback information and the target description information to generate and output auxiliary filling information; if the intent type contains a specified type, triggering multiple rounds of dialogue to obtain filling content, and inputting the filling content into the target form item.

Images