Data processing method and device and electronic equipment

Abstract

The invention discloses a data processing method and device and electronic equipment, and the method comprises the steps: obtaining a target operation for a target image when a display screen of the electronic equipment outputs the target image; determining target information based on the target operation, wherein the target information can respond to an operation intention of the target operation; and outputting the target information.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and particularly to a data processing method, apparatus, and electronic device. Background Art

[0002] Currently, when operating on an image output by an electronic device, complex interaction between the user and the electronic device is required to obtain the information that the user wants to know through the image, and this process is complex in operation. Summary of the Invention

[0003] In view of this, this application provides a data processing method, apparatus, and electronic device, and its specific solutions are as follows:

[0004] A data processing method includes:

[0005] When a target image is output on a display screen of an electronic device, obtain a target operation for the target image;

[0006] Determine target information based on the target operation, where the target information can respond to the operation intention of the target operation;

[0007] Output the target information.

[0008] Further, the determining the target information based on the target operation includes:

[0009] In response to the target image having a first presentation mode, obtain the position information of the target operation on the target image, and determine the output information corresponding to the position information on the target image, where the first presentation mode makes the image not have elements that can be operated independently;

[0010] In response to the target image having a second presentation mode, obtain the output information displayed in response to the target operation, where the second presentation mode makes the image have at least one element that can be operated independently;

[0011] Determine the target information based on the output information.

[0012] Further, the determining the target information based on the target operation includes:

[0013] Analyze the target image to determine the information included in the target image;

[0014] Determine the target information based on the target operation and the information included in the target image.

[0015] Further, the determining the target information based on the target operation and the information included in the target image includes:

[0016] If the information included in the target image includes a target object, query the information of the target object;

[0017] Determine the target information based on the target operation, the information included in the target image, and the information of the target object.

[0018] Further, if the target image is an image capable of representing a data trend, the determining the target information based on the target operation includes:

[0019] Analyze the target image and the target operation for the target image to determine multiple fields and the data corresponding to each field respectively;

[0020] Fill the multiple fields and the data corresponding to each field into the data structure corresponding to the target image, and the multiple fields included in the data structure include at least two of the following: the chart type corresponding to the data structure, the information of the operation point corresponding to the target operation, the keyword of the target image, the key information of the data structure;

[0021] Determine the target information based on the data structure.

[0022] Further, the determining the target information based on the data structure includes:

[0023] Input the data structure into the prompt word generation model;

[0024] Obtain at least one candidate question output by the prompt word generation model based on the data structure;

[0025] In response to a selection operation on the at least one candidate question, input the target question selected based on the selection operation into the large model;

[0026] Obtain the target information output by the large model based on the target question.

[0027] Further, the determining the target information based on the target operation includes:

[0028] Invoke the target model based on the target operation, and determine the target information through the target model.

[0029] Further, the outputting the target information includes at least one of the following:

[0030] Output the target information through the output window of the target model;

[0031] Output the target information at the target position of the display screen, and the target position of the display screen is at least related to the position where the display screen outputs the target image.

[0032] A data processing device, comprising:

[0033] An obtaining unit, configured to obtain a target operation for the target image when the target image is output on a display screen of an electronic device;

[0034] A determining unit, configured to determine target information based on the target operation, where the target information can respond to the operation intention of the target operation;

[0035] An output unit, configured to output the target information.

[0036] An electronic device, comprising:

[0037] A display screen;

[0038] A processor, configured to obtain a target operation for the target image when the target image is output on the display screen; determine target information based on the target operation, where the target information can respond to the operation intention of the target operation; and output the target information through the display screen. Description of the Drawings

[0039] To more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the following will briefly introduce the drawings required for use in the description of the embodiments or the related art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0040] Figure 1 It is a flowchart of a data processing method disclosed in an embodiment of the present application;

[0041] Figure 2 It is a flowchart of a data processing method disclosed in an embodiment of the present application;

[0042] Figure 3 It is a schematic diagram of the position of a target operation on a target image disclosed in an embodiment of the present application;

[0043] Figure 4 It is a schematic diagram of a bar chart of the monthly output of a certain device in the first quarter of a certain year disclosed in an embodiment of the present application;

[0044] Figure 5 It is a flowchart of a data processing method disclosed in an embodiment of the present application;

[0045] Figure 6 It is a flowchart of a data processing method disclosed in an embodiment of the present application;

[0046] Figure 7 It is a flowchart of a data processing method disclosed in an embodiment of the present application;

[0047] Figure 8 Schematic diagram of a display interface of a data processing method disclosed in an embodiment of the present application;

[0048] Figure 9 Schematic diagram of the structure of a data processing device disclosed in an embodiment of the present application;

[0049] Figure 10 Schematic diagram of the structure of an electronic device disclosed in an embodiment of the present application. Detailed implementation manners

[0050] The embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application. The terms used in the implementation manners part of the present application are only used to explain the specific embodiments of the present application, rather than to limit the present application.

[0051] The embodiments of the present application will be described below with reference to the accompanying drawings. Those of ordinary skill in the art will know that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are equally applicable to similar technical problems.

[0052] The terms "first", "second", etc. in the specification, claims and above-mentioned drawings of the present application are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that such terms can be interchanged under appropriate circumstances, which is only a way of distinguishing objects with the same attributes when describing the embodiments of the present application. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion, so that a process, method, system, product or device including a series of units does not have to be limited to those units, but may include other units not clearly listed or inherent to these process, method, product or device.

[0053] The present application discloses a data processing method, and its flowchart is as Figure 1 shown, including:

[0054] Step S11, when a target image is output on the display screen of the electronic device, obtain a target operation for the target image;

[0055] Step S12, determine target information based on the target operation, and the target information can respond to the operation intention of the target operation;

[0056] Step S13, output the target information.

[0057] Currently, when operating on an image output by an electronic device, a user needs to perform complex interactions with the electronic device to obtain the information that the user wants to know through the image, and this process is complex in operation.

[0058] Based on this, in this solution, when obtaining a target operation for a target image displayed on a display screen, the target operation is analyzed to determine target information of an operation intention that can respond to the target operation, and the target information is output, so that an electronic device can output, based on the target operation for the target image displayed, the target information of the operation intention that responds to the target operation, without the need for complex interaction between the user and the electronic device, nor the need for the user to manually input text for the image, thus improving the user experience.

[0059] When the electronic device outputs information through the display screen, the information output by the display screen may only include the target image. For example, the display screen of the electronic device displays the target image full screen. It may also include both the target image and other information. For example, the display screen of the electronic device does not display the target image full screen, and other information is also displayed in the area where the target image is not displayed on the display screen. The other information may be: display controls on the display screen, or other images or interfaces displayed on the display screen, etc.

[0060] When the information output by the display screen includes the target image, operations can be performed on the target image. The operations on the target image involved in this embodiment are not operations performed on the editing controls in the program that displays the image. For example, operations performed on switching controls for switching to the next image or the previous image, and operations performed on editing controls such as rotating and editing the target image, but are target operations such as clicking and swiping performed on the content of the target image itself.

[0061] Among them, the target operation performed on the target image can be: performing a target operation on the target image through the display screen. For example, if the display screen is a touch display screen, directly perform operations such as clicking or swiping on the target image displayed on the touch display screen; or, it can also be: performing operations such as clicking or swiping on the target image displayed on the display screen through an external device such as a mouse or a keyboard.

[0062] The electronic device obtains a target operation for the target image, that is, obtains a target operation for the target image through the display screen of the electronic device, or obtains a target operation for the target image through an external device of the electronic device.

[0063] After obtaining the target operation for the target image, the target operation is analyzed to determine the operation intention of the user performing the target operation.

[0064] For example: If the user clicks on a specific point in the target image, and the specific point corresponds to an English word in the target image, then the operation of clicking on the specific point can be used to determine that the user wants to recognize or translate the English word, and the recognition or translation of the English word is determined as the operation intention.

[0065] After analyzing the target operation and determining the operation intention of the user to perform the target operation, it is necessary to determine the target information that responds to this operation intention and directly output this target information. In this process, only the target operation needs to be performed on the target image, and no other operations are required. Thus, based on the target operation on the target image, the target information that responds to the operation intention of this target operation can be directly output. Without manually inputting the text corresponding to the operation intention, the target information that responds to the operation intention of this target operation can be directly obtained, simplifying the user operation and improving the user experience.

[0066] The target information that responds to the operation intention can be: the information output when the electronic device responds to this operation intention. For example, if the operation intention is to identify or translate a specific English word, then the target information that responds to this operation intention is: the result of identifying or translating this specific English word. Then, without manually inputting the operation intention, the target information that responds to the operation intention can be directly output, thus achieving the purpose of simplifying the user operation.

[0067] Here, it should be noted that the target image can be an image without elements that can be operated independently, such as an image in an album; or it can be an image with elements that can be operated independently, such as the interface of a certain application or control.

[0068] For example: The target image is a data trend graph. This data trend graph is an image without elements that can be operated independently, or this data trend graph is an image with elements that can be operated independently. When performing a target operation on this data trend graph, it can be: sliding at the position of July - October in this data trend graph. Then the operation intention of this target operation can be: analyzing the data from July - October in this data trend graph, and the target information that responds to this operation intention can be: the analysis result of analyzing the data from July - October in this data trend graph, and directly output this analysis result.

[0069] The data processing method disclosed in this embodiment obtains a target operation for the target image when the target image is output on the display screen of the electronic device; determines the target information based on the target operation, and the target information can respond to the operation intention of the target operation; outputs the target information. In this solution, when the target image is output on the display screen, if a target operation for the target image is obtained, the target operation is not directly executed. Instead, it is necessary to determine the operation intention of this target operation and determine the target information used to respond to this operation intention, and output this target information. Thus, when performing a target operation on the target image output on the display screen, the operation intention of this operation can be analyzed, and the target information that responds to this operation intention can be output, thereby ensuring that the response to the operation intention of this target operation is directly output based on the target operation, without the user having to perform multiple operations or complex operations, simplifying the user operation and improving the user experience.

[0070] This embodiment discloses a data processing method, and its flowchart is as Figure 2 shown, including:

[0071] Step S21: When a target image is output on the display screen of the electronic device, obtain a target operation for the target image;

[0072] Step S22: In response to the target image having a first presentation mode, obtain the position information of the target operation on the target image, and determine the output information corresponding to the position information on the target image. The first presentation mode enables no element that can be operated independently;

[0073] Step S23: Determine target information based on the output information, and the target information can respond to the operation intention of the target operation;

[0074] Step S24: Output the target information.

[0075] When a target image is output on the display screen of the electronic device, if a target operation for the target image is obtained, it is necessary to first determine whether the target image has an element that can be operated independently.

[0076] If the target image has an element that can be operated independently, whether the element that can be operated independently is one or more, it can be determined that the target image has a second presentation mode; if the target image does not have an element that can be operated independently, it can be determined that the target image has a first presentation mode. When the target image has different presentation modes, the method of determining target information based on the target operation is different.

[0077] Specifically, if the target image has a first presentation mode, that is, the target image does not have an element that can be operated independently. At this time, the target image can be an image in the album or an image on the network, etc.

[0078] When obtaining a target operation for the target image, since the target image does not have an element that can be operated independently, therefore, the target operation cannot change the target image. Therefore, it is necessary to determine the target information based on the position of the target operation on the target image.

[0079] First, determine the position information of the target operation for the target image on the target image, rather than the position information of the target operation relative to the display screen. Specifically, it can be the coordinates of the target operation relative to the target image. As Figure 3 shown, it is a schematic diagram of the position of the target operation on the target image, including: display screen 31, target image 32 displayed on the display screen 31, target operation 33, determine the position of the target operation 33 on the target image 32, Figure 3If the target operation 33 is to slide on the target image 32, then when determining the position of the target operation 33 on the target image 32, it is necessary to determine the position of the starting point and the ending point of the target operation 33. Among them, the position of the starting point can be the coordinates of the starting point on the target image, and the position of the ending point can be the coordinates of the ending point on the target image.

[0080] After determining the position information of the target operation on the target image, it is necessary to further determine the information corresponding to this position information on the target image, that is, the output information. If the position information of the target operation on the target image is the coordinates of the target operation on the target image, then it is necessary to determine the information at this coordinate position on the target image, that is, what content corresponds to this coordinate on the target image. As Figure 4 shown, it is a bar chart of the monthly output of a certain device in the first quarter of a certain year. Among them, the output in January is a, the output in February is b, and the output in March is c. The target operation is presented as 41 on the bar chart, which is a sliding operation on the bar chart. This sliding operation corresponds to January and February in the bar chart. Then, it can be determined that the output information corresponding to this position information on the target image is: the output in January and February.

[0081] After determining the output information corresponding to this position information on the target image, the target information can be determined based on this output information. Specifically, after determining the output information, the operation intention of executing this target operation can be determined based on this output information, and further, the target information corresponding to this operation intention can be determined based on the operation intention.

[0082] Continue to take Figure 4 as an example to illustrate. After determining that the output information is the output in January and February, determine the operation intention, which can be: determine the change rate of the output in January and February. Then, based on this operation intention and the bar chart, analyze the change rate of the output in January and February, and output the analyzed result as the target information; or, after determining that the output information is the output in January and February, when determining the operation intention, continue to analyze the target operation, determine the trend of the target operation, that is, determine that the target operation is carried out according to the change trend of the output in January and February. Then, based on the fact that the determined target operation is carried out according to the change trend of the output in January and February and the output information, determine the operation intention. At this time, the operation intention can be: analyze the reason for the decrease in the output in January and February, and analyze and determine the target information corresponding to this operation intention, that is, output the analyzed reason for the decrease in the output in January and February as the target information.

[0083] For another example: The target image is a screenshot of the interface when the device crashes. There is an "exclamation mark" on this screenshot to indicate that the device has crashed. Then, determine the position information of the target operation on the target image, and determine the output information corresponding to the position information on the target image. If the "exclamation mark" corresponds to the position information on the target image, then determine that the operation intention is to analyze the cause of the device crash, and determine that the target information is the specific cause of the device crash obtained by analyzing according to the operation intention.

[0084] Furthermore, if the target image has a second presentation mode, that is, the target image has at least one element that can be operated independently, at this time, the target image can be an interface with operation controls.

[0085] If the target image with the second presentation mode receives the target operation, determine whether the target operation is performed on an element that can be operated independently on the target image. If so, in response to the target operation performed on this element, display the output information, and the displayed output information is the information displayed after the element performs the target operation. For example, if the target operation is directed at the "query reference materials" control on the target image, then in response to the target operation performed on this control, directly display the process and results of querying reference materials and use them as the output information; if the target operation is performed on an element that cannot be operated independently on the target image, then determine according to the position information of the target operation on the target image, that is, determine the position information of the target operation on the target image, and determine the output information corresponding to the position information on the target image.

[0086] The data processing method disclosed in this embodiment, when obtaining the target operation for the target image output on the display screen, if the target image does not have an element that can be operated independently, it is necessary to determine the position information of the target operation on the target image, and further determine the output information corresponding to this position information on the target image, and determine and output the target information based on the output information. When the target image cannot be operated independently, it is necessary to determine the position of the target operation on the target image, and further determine the information on the target image corresponding to this position, and determine the target information based on this, ensuring the accuracy of the determination of the target information and improving the user experience.

[0087] This embodiment discloses a data processing method, and its flowchart is as Figure 5 shown, including:

[0088] Step S51: When a target image is output on the display screen of the electronic device, obtain the target operation for the target image;

[0089] Step S52: Analyze the target image to determine the information included in the target image;

[0090] Step S53: Determine target information based on the target operation and the information included in the target image, where the target information can respond to the operation intention of the target operation;

[0091] Step S54: Output the target information.

[0092] When a target image is displayed on the display screen of an electronic device, if a target operation for the target image is obtained, at this time, it is necessary to first analyze the target image to determine the information included in the target image, so as to determine the target information based on the information included in the image and the target operation, and output the target information.

[0093] Among them, determining the information included in the target image can be: analyzing the target image to determine what objects or data are included in the target image. For example, determining that there are two dogs and three trees in the target image; for another example, determining that there is a laptop computer in the target image, and the brand and model of the laptop computer can be determined from the target image; for another example, determining that there is a pie chart in the target image, and the pie chart records the relevant data of different models of laptop computers sold by store A in a certain month and year; for another example, determining that there is a piece of text in the target image, and the text includes a title and specific text content, and analyzing the meaning of the title and specific text content, etc.

[0094] In addition, when determining the information included in the target image, it is not only necessary to determine what objects or data are included in the target image, but also necessary to further analyze the format, characteristics, etc. of the objects or data included in the target image.

[0095] For example, if the target image is a line chart, when determining the information included in the target image, it is not only necessary to determine the title of the line chart, the data in the line chart, but also necessary to determine the colors, font sizes, etc. of the data in the line chart, and it is also necessary to determine whether there are annotations, legends, etc. in the line chart. If there are annotations, legends, etc., it is also necessary to determine the content of the annotations, legends, etc.; in addition, it is also necessary to determine the coordinate system where the line chart is located, the meaning of the coordinate axes, etc., to ensure the integrity of the information included in the determined target image.

[0096] After analyzing the information included in the target image, determine the target information based on the target operation and the information included in the target image, that is: since the target operation is executed on the target image, therefore, combine the information included in the target image with the target operation and analyze them together to determine the operation intention, and determine the target information based on the operation intention, so as to output the target information.

[0097] For the same target image, different operation intents and target information can be determined based on different target operations; correspondingly, for the same target operation, different operation intents and target information can also be determined based on different target images; in addition, for the same target image and the same target operation, even if the position on the target image where the target operation acts is different, the determined operation intents and target information are also different.

[0098] For example: The target image includes two dogs and three trees. If the operation performed on this target image is a click operation, the operation intent of this click operation may be: analyzing the breeds of the objects in the target image corresponding to the click operation; if the operation performed on this target operation is a swipe operation, the operation intent of this swipe operation may be: analyzing the differences between the two dogs in the target image, or analyzing the scene corresponding to the target image, such as: determining the time, location, etc. corresponding to the target image by analyzing the background, light, and trees in the target image.

[0099] Another example: The target image is a pie chart, and the pie chart includes data related to different models of laptops sold by store A in a certain month and year. If the operation performed on this target image is a click operation, when the position corresponding to the click operation on the pie chart in the target image is position a, the operation intent is: analyzing the sales situation of the laptop of model 1 corresponding to position a in this pie chart; when the position corresponding to the click operation on the pie chart in the target image is position b, the operation intent is: analyzing the sales situation of the laptop of model 2 corresponding to position b in this pie chart; if the click operation corresponds to the title of the pie chart in the target image, then analyze the data of different models of laptops sold by this store in a certain month and year, etc.

[0100] Another example: The target image is a line chart, and the line chart includes the sales data of a certain model of mobile phone in different months of a certain year. If the operation performed on this target image is a swipe operation, when the position corresponding to the swipe operation on the line chart is the position where January - March is located, the operation intent is: analyzing the change in the sales data of this model of mobile phone from January to March; when the position corresponding to the swipe operation on the line chart is the position where February - July is located, the operation intent is: analyzing the change in the sales data of this model of mobile phone from February to July, etc.

[0101] Therefore, when determining the target information, it is necessary to first determine the information included in the target image, and after determining the information included in the target image, determine the operation intent based on the target operation for the target image and the information included in the target image, and further determine the target information corresponding to this operation intent to ensure the accuracy of the determination of the target information.

[0102] In the data processing method disclosed in this embodiment, when obtaining a target operation for a target image output by a display screen, the target image is first analyzed to determine the information included in the target image, and then the target information of the operation intention that can respond to the target operation is determined based on the target operation and the information included in the target image, so as to output the target information. This realizes directly outputting the target information of the operation intention that can respond to the target operation based on the target operation for the target image, without manually inputting the operation intention and then determining and outputting the target information in response to the operation intention, simplifies the user's operation, and improves the user experience.

[0103] Further, in the data processing method disclosed in this embodiment, determining the target information based on the target operation and the information included in the target image may be:

[0104] If the information included in the target information includes a target object, query the information of the target object; determine the target information based on the target information, the information included in the target image, and the information of the target object.

[0105] The information included in the target image may be: text information, for example, the target image includes at least one piece of text content, or non-text information, such as images of people, animals, or tables, etc. Whether the target image includes text information, non-text information, or both text information and non-text information, it is necessary to determine the objects included in the information included in the target image.

[0106] That is, if the target image includes text information, then this text information may be information describing a certain object. At this time, it is necessary to determine the target object included in the information included in the target image. For example, if the target image includes a text description of product B, then product B is determined as the target object; another example, if the target image includes an introduction to a certain person, then this person is determined as the target object, etc.

[0107] If the target image includes non-text information, then this non-text information may be information explaining or demonstrating a certain object. At this time, it is necessary to determine the target object included in the information included in the target image. For example, if the target image includes an image of a chair, then the chair is determined as the target object; another example, if the target image includes a production data table of product C, then product C is determined as the target object, etc.

[0108] After determining the target object, it is also necessary to query the information of the target object. At this time, querying the information of the target object is not to query from the target image, but to query the information related to the target object that is not shown on the target image, such as querying the information related to the target image locally, or querying the information related to the target image at the network end.

[0109] After information related to the target image is queried, it is jointly analyzed with the information included in the target image and the target operation to determine the operation intention corresponding to the target operation, and further determine the target information, so as to ensure that the determined operation intention is more accurate and comprehensive, and thus ensure the accuracy of the output target information.

[0110] For example, if the target image includes a text description of product B, then product B is determined as the target object, information related to product B is queried from the network, and the queried information is jointly analyzed with the information included in the target image and the target operation to determine the operation intention and further determine the target information.

[0111] This embodiment discloses a data processing method, and its flowchart is as Figure 6 shown, including:

[0112] Step S61: When a target image is output on the display screen of the electronic device, obtain a target operation for the target image, where the target image is an image that can represent a data trend;

[0113] Step S62: Analyze the target image and the target operation for the target image to determine multiple fields and the data corresponding to each field respectively;

[0114] Step S63: Fill the multiple fields and the data corresponding to each field into the data structure corresponding to the target image. The multiple fields included in the data structure include at least two of the following: the chart type corresponding to the data structure, the information of the operation point corresponding to the target operation, the keywords of the target image, and the keywords of the data structure;

[0115] Step S64: Determine the target information based on the data structure, where the target information can respond to the operation intention of the target operation;

[0116] Step S65: Output the target information.

[0117] The target image can be various types of images, such as: landscape images, portrait images, table images, text images, images with elements that can be individually operated, images without elements that can be individually operated, etc.

[0118] In this embodiment, the target image is an image that can represent a data trend, which can be a line chart, a pie chart, a bar chart, etc. Among them, the target image that can represent a data trend can be an image with elements that can be individually operated, or an image without elements that can be individually operated.

[0119] When a target image capable of representing a data trend is output on the display screen of an electronic device, if a target operation for the target image is obtained, the target image and the target operation need to be analyzed to determine the operation intention of the target operation, and to determine the target information in response to the operation intention, and output the target information, so as to achieve that the target information in response to the operation intention can be output only through the target operation for the target image, without the user manually inputting the intention, simplifying the operation process and improving the user experience.

[0120] Specifically, when the target image is an image capable of representing a data trend, a corresponding data structure can be constructed through the target image and the target operation for the target image, so as to determine the operation intention based on the data structure, and determine and output the target information based on the operation intention.

[0121] The data structure can include multiple fixed fields, such as: the chart type corresponding to the data structure, the information of the operation point corresponding to the target operation, the keywords of the target image, the key information of the data structure, etc. The target image and the target operation for the target image can be analyzed to determine the data corresponding to each field, and these data are filled into the corresponding data structure to form the filled data structure, so as to use the filled data structure to determine the target information.

[0122] When a target operation for the target image is obtained, the target image and the target operation for the target image are analyzed to determine multiple fields and the data corresponding to each field. First, the overall information of the target image can be obtained, such as: the theme name of the target image, the keywords of the target image, the chart type, etc. information, and then the information determined based on the target operation in the target image can be obtained, such as: which data in the target image does the target operation correspond to, that is, the information of the operation point on the target image corresponding to the target operation, etc.; in addition, the various information obtained above can be analyzed and sorted to infer certain specific data, such as: the key information of the data structure, etc. Then, determine the fields corresponding to the information obtained, and fill them into the corresponding data structure according to their corresponding fields.

[0123] For example: the obtained theme name of the target image can correspond to the field of "keywords of the target image", and the information of the operation point on the target image corresponding to the obtained target operation can correspond to the field of "information of the operation point corresponding to the target operation"; in addition, keywords can be inferred based on the obtained information, and the inferred keywords are determined to correspond to the field of "key information of the data structure".

[0124] For example: By analyzing the target image and the target operation on the target image, it is determined that the chart type of the data structure is a line chart. The points on the line chart in the data structure are determined by the information of the operation points of the target operation on the target image, etc., so as to obtain a line chart filled with various data points.

[0125] If the chart type of the data structure is a line chart, it can show the change trend of the data. The points on the line chart have abscissa and ordinate information. The specific trend can be calculated between every two operation points of the target operation on the line chart. If the number of operation points of the target operation on the line chart is greater than 3, the overall trend of the line chart can be determined based on the operation points on the line chart.

[0126] If the chart type of the data structure is a pie chart, it can show the proportional relationship between different data in the chart. The proportion occupied by the area where the operation point of the target operation is located on the pie chart can be calculated. In addition, the pie chart can also show the proportional distribution of specific time.

[0127] If the chart type of the data structure is a bar chart, it can show the trend of the data and compare the differences in the data. Each bar on the bar chart has abscissa and ordinate information, and each bar has a clear representativeness, such as: product name, time name, etc. For bars of the same type, there are different trend changes on different horizontal or vertical axes.

[0128] Based on the target image and the target operation on the target image, data corresponding to multiple fields can be obtained, but it does not necessarily obtain data corresponding to each field in the data structure. Only the data corresponding to the obtained multiple fields are filled into the corresponding fields of the data structure, so as to obtain a filled data structure.

[0129] After determining the data corresponding to each field using the target image that can characterize the data trend, the determined data corresponding to each field is filled into the corresponding data structure, so as to analyze the filled data structure, determine the operation intention of the target operation, and further analyze and determine the target information that can respond to the operation intention.

[0130] Specifically, the data structure can be in the form of code. The filled data structure can be as follows:

[0131] {

[0132] "chart": "line chart",

[0133] "points": [{

[0134] "date": "20240701",

[0135] "value": "96080",

[0136] "product": "Thinkpad X1 Carbon"

[0137] },

[0138] {

[0139] "date": "20241001",

[0140] "value": "270000",

[0141] "product": "Thinkpad X1 Carbon"

[0142] }],

[0143] "title": "MAU(Last_6_Month)",

[0144] "keywords":

[0145] "Trend",

[0146] "MAU",

[0147] "Thinkpad X1 Carbon",

[0148] "Growth",

[0149] "Detailed Distribution"

[0151] }

[0152] ​The data processing method disclosed in this embodiment obtains a target operation for a target image when the target image is output on the display screen of an electronic device, where the target image is an image capable of representing a data trend; analyzes the target image and the target operation for the target image to determine multiple fields and the data corresponding to each field; fills the multiple fields and the data corresponding to each field into the data structure corresponding to the target image, and the multiple fields included in the data structure include at least two of the following: the chart type corresponding to the data structure, the information of the operation point corresponding to the target operation, the keyword of the target image, the keyword of the data structure; determines target information based on the data structure, and the target information can respond to the operation intention of the target operation; outputs the target information. In this solution, when the target image is an image capable of representing a data trend, it is necessary to fill the data structure based on the target image and the target operation, so as to determine the target information, so as to realize the conversion of the image representing the data trend into a data structure and output the target information responding to the operation intention of the target operation, so as to ensure that the target information determined based on the operation intention can be output only based on the target operation, simplify the user operation, and improve the user experience.

[0153] This embodiment discloses a data processing method, and its flowchart is as Figure 7 shown, including:

[0154] Step S71, when a target image is output on the display screen of an electronic device, obtain a target operation for the target image, where the target image is an image capable of representing a data trend;

[0155] Step S72, analyze the target image and the target operation for the target image to determine multiple fields and the data corresponding to each field;

[0156] Step S73, fill the multiple fields and the data corresponding to each field into the data structure corresponding to the target image;

[0157] Step S74, input the data structure into the prompt word generation model;

[0158] Step S75, obtain at least one candidate question output by the prompt word generation model based on the data structure;

[0159] Step S76, in response to a selection operation on at least one candidate question, input the target question selected based on the selection operation into the large model;

[0160] Step S77, obtain target information output by the large model based on the target question, where the target information can respond to the operation intention of the target operation;

[0161] Step S78, output the target information.

[0162] If the target image is an image that can represent data trends, after obtaining the target operation for the target image, the target image and the target operation can be analyzed to determine the data for different fields therefrom, and the determined data for different fields can be filled into the data structure according to the corresponding fields, so as to obtain the operation intention of the target operation based on the data structure and further obtain the target information that can respond to the operation intention.

[0163] Among them, obtaining the operation intention of the target operation based on the data structure and further obtaining the target information that can respond to the operation intention can be specifically as follows: First, input the data structure into the prompt word generation model. After obtaining at least one candidate question output by the prompt word generation model based on the data structure, display it to the user and obtain the user's selection operation for the at least one candidate question; then input the target question selected by the selection operation into the large model so that the large model can output the target information that responds to the operation intention of the target operation.

[0164] The prompt word generation model is used to generate at least one candidate question according to the data structure. Therefore, after obtaining the filled data structure, directly input it into the prompt word generation model. The prompt word generation model is used to analyze the data structure. Since the data structure is generated based on the target image and the target operation for the target image, when the prompt word generation model analyzes the data structure, it can analyze and obtain the operation intention of the user performing the target operation on the target image. The operation intention is the candidate question, that is: what question does the user want to ask by performing the target operation on the target image. It is analyzed by the prompt word generation model, and the analyzed candidate question is output as the operation intention.

[0165] The prompt word generation model needs to have a certain description to generate prompt words. The prompt word generation model generates prompt words according to the input description. In this embodiment, the data structure is used as the description input to the prompt word generation model, so that the prompt word generation model generates prompt words, that is, multiple candidate questions, based on the data structure. And the data structure in this embodiment is obtained through the target operation performed by the user on the target image, that is, directly convert the target operation on the target image into a data structure, so that the prompt word generation model can generate multiple candidate questions based on the data structure. In this process, through the target operation performed on the target image and based on the target operation to generate a data structure, there is no need for the user to perform text input for the description input to the prompt word generation model. By replacing the description input to the prompt word generation model with the operation on the image, without the user inputting the description of the prompt word generation model through text, only by operating on the target image, multiple candidate questions output by the prompt word generation model can be obtained, which simplifies the user operation.

[0166] The prompt generation model analyzes the data structure and may obtain multiple candidate questions. Each candidate question serves as an operation intention. When there are multiple candidate questions, there are multiple corresponding operation intentions. For example: The target operation is a sliding operation on a line chart. The starting point of the sliding operation is a, and the ending point is b. Based on this target image and target operation, the data structure is obtained and input into the prompt generation model. The prompt generation model analyzes it and can obtain candidate question 1: Analyze the data change trend from point a to point b, candidate question 2: Why does such a change trend occur from point a to point b, etc.

[0167] However, the prompt generation model cannot determine which candidate question is the operation intention of the user to perform the target operation. Therefore, at least one candidate question needs to be output so that the user can perform a selection operation on at least one candidate question. Since at least one candidate question is the candidate operation intention analyzed by the prompt generation model based on the data structure, after the user performs a selection operation on at least one candidate question, the user selects the target question from at least one candidate question, and this target question is the operation intention of the user to perform the target operation.

[0168] In addition, if the candidate questions output by the prompt generation model based on the data structure are multiple, then multiple candidate questions need to be output according to the above content so that the user can select one of them as the target question; in addition, it can also be: when the candidate questions output by the prompt generation model based on the data structure are multiple, the prompt generation model not only outputs multiple candidate questions but also performs a priority ranking on the multiple candidate questions it outputs, that is, which candidate question has the highest priority and which has the lowest priority among the multiple candidate questions. At this time, the candidate question with the highest priority can be directly determined as the target question; if the candidate question output by the prompt generation model based on the data structure is only one, then the candidate question output by the prompt generation model is directly determined as the target question.

[0169] After determining the target question, it is input into the large model, and the large model responds to the target question, and the information obtained is the target information, that is, the target information that can respond to the operation intention of the target operation is obtained.

[0170] That is, first use the prompt generation model to determine at least one candidate question. After selecting the target question from at least one candidate question, use the large model to determine the target information that can respond to the target question, so as to determine the target information of the operation intention that can respond to the target operation through two models. The user only needs to perform the target operation on the target image and select the target question from at least one candidate question, without having to input text and images into the large model in a cumbersome manner, and can obtain the target information they want, simplifying the user operation.

[0171] Among them, the large model involved in this embodiment can be: RAG (Retrieval-Augmented Generation) model, which is a technology that combines information retrieval and generation models, aiming to enhance the performance of the large language model LLM through an external knowledge base.

[0172] For the data processing method disclosed in this embodiment, when the target image is an image that can represent a data trend, if a target operation for the target image is obtained, it is necessary to determine the data corresponding to each field based on the target image and the target operation, and fill the obtained data corresponding to each field into the data structure, so as to input the data structure into the prompt word generation model, thereby obtaining at least one candidate question output by the prompt word generation model. After the user selects a candidate question, the selected target question can be input into the large model to obtain target information and directly output the target information. The complete output process of the above target information only requires performing the target operation on the target image and selecting the target question from at least one candidate question, then the target information that can respond to the operation intention of the target operation can be obtained and directly output, without the user manually inputting the operation intention, which simplifies the user operation and improves the user experience.

[0173] Furthermore, the data processing method disclosed in this embodiment can also be: when the target image is output on the display screen of the electronic device and a target operation for the target image is obtained, directly start the target model based on the target operation, and determine the target information of the operation intention that can respond to the target operation through the target model.

[0174] That is, in this embodiment, the system based on the data processing method disclosed in this embodiment is no longer used to analyze the target image and the target operation to determine the data structure, nor are the prompt word generation model and the large model used to perform different steps respectively to obtain the target information of the operation intention that can finally respond to the target operation. Instead, after it is determined that there is a target operation for the target image, the target model is directly used to analyze the target image and the target operation to obtain the target information of the operation intention that can respond to the target operation, further reducing the user's operation and improving the user experience.

[0175] Among them, after obtaining the target operation for the target image, starting the target model means that when the target image is output on the display screen of the electronic device, the target model is in an unstarted state. Only after obtaining the target operation, the target model will be called and started, and the target model will be used to analyze the target image and the target operation to determine the target information of the operation intention that can respond to the target operation and output it.

[0176] Furthermore, in the data processing method disclosed in this embodiment, outputting the target information may include at least one of the following:

[0177] Output the target information through the output window of the target model; output the target information at the target position on the display screen, and the target position on the display screen is at least related to the position where the display screen outputs the target image.

[0178] After starting the target model based on the target operation and determining the target information of the operation intention that can respond to the target operation by using the target model, the target information can be directly output by using the output window of the target model. That is, after starting the target model, the target model is displayed on the display screen of the electronic device, and after the target model analyzes and obtains the target information, the target information is directly displayed in the output window of the target model.

[0179] Alternatively, it can also be: after starting the target model based on the target operation, the interface of the target model may not be displayed on the display screen, but the target model runs in the background, and after the target model outputs the target information, the target information is output at the target position on the display screen, and the target position is related to the position of the target image on the display screen. For example, the target information is displayed within a certain distance of the target image. More specifically, the target information can be displayed in the upper right corner of the target image, and the target information can be displayed in the form of an active window, or in the form of a label; or, the target information is displayed within a certain distance of the position of the target operation on the target image, etc.

[0180] The schematic diagram of the display interface of the data processing method disclosed in this embodiment can be as Figure 8 shown Figure 8 In the schematic diagram of the display interface shown, the target image is directly displayed on the target model (the target image can also be directly displayed on other application programs, and this is only for illustrative purposes here). At this time, the target model can be input to make the target model display the target image. Taking the target model as the target AI assistant as an example, as Figure 8 shown in, user A inputs a request on the target model "Please show the MAU of XX model notebooks in the past 6 months", where MAU is the monthly user activity. At this time, the target model B responds to this request and displays the line chart 81 (of course, the process of the user inputting a request on the target model and the target model responding to this request does not have to be implemented on the target model, and subsequent operations can also be directly performed on a certain displayed image. This is only for illustrative purposes here). After that, the user performs a target operation on the line chart 81, such as sliding on the line of the line chart, then the target model analyzes this target operation and obtains at least one candidate question (where, analyzing the target operation, obtaining the filled data structure, and then inputting the data structure into the prompt word generation model. For these steps, since Figure 8It is implemented directly on a target model, so there is no step of inputting the data structure into the prompt word generation model. Then, the filled data structure will not be displayed, but the analysis of the target operation will be directly performed in the background to obtain the data structure, and at least one candidate question will be generated using the obtained data structure (these steps). At least one candidate question 82 obtained by the target model can be displayed on the target image on the display interface of the target model; afterwards, the user selects a candidate question from at least one candidate question 82 as the target question, which is presented as the user inputting the target question on the display interface, and the target model outputs the target information 83 based on the target question.

[0181] This embodiment discloses a data processing device, and its structural schematic diagram is as Figure 9 shown, including:

[0182] An acquisition unit 91, a determination unit 92 and an output unit 93.

[0183] Among them, the acquisition unit 91 is used to acquire a target operation for the target image when the target image is output on the display screen of the electronic device;

[0184] The determination unit 92 is used to determine the target information based on the target operation, and the target information can respond to the operation intention of the target operation;

[0185] The output unit 93 is used to output the target information.

[0186] Furthermore, the determination unit is used to:

[0187] In response to the target image having a first presentation mode, obtain the position information of the target operation on the target image, and determine the output information corresponding to the position information on the target image. The first presentation mode makes the image not have elements that can be operated independently; in response to the target image having a second presentation mode, obtain the output information displayed in response to the target operation. The second presentation mode makes the image have at least one element that can be operated independently; determine the target information based on the output information.

[0188] Furthermore, the determination unit is used to:

[0189] Analyze the target image to determine the information included in the target image; determine the target information based on the target operation and the information included in the target image.

[0190] Furthermore, the determination unit is used to:

[0191] If the information included in the target image includes a target object, query the information of the target object; determine the target information based on the target operation, the information included in the target image and the information of the target object.

[0192] Further, if the target image is an image capable of characterizing data trends, further, the determination unit is configured to:

[0193] Analyze the target image and the target operation for the target image to determine a plurality of fields and the data corresponding to each field; fill the plurality of fields and the data corresponding to each field into the data structure corresponding to the target image, and the plurality of fields included in the data structure include at least two of the following: the chart type corresponding to the data structure, the information of the operation point corresponding to the target operation, the keyword of the target image, and the key information of the data structure; determine the target information based on the data structure.

[0194] Further, the determination unit is configured to:

[0195] Input the data structure into the prompt word generation model; obtain at least one candidate question output by the prompt word generation model based on the data structure; in response to the selection operation on the at least one candidate question, input the target question selected based on the selection operation into the large model; obtain the target information output by the large model based on the target question.

[0196] Further, the determination unit is configured to:

[0197] Activate the target model based on the target operation, and determine the target information through the target model.

[0198] Further, the output unit:

[0199] Output the target information through the output window of the target model;

[0200] Or, output the target information at the target position of the display screen, and the target position of the display screen is at least related to the position where the display screen outputs the target image.

[0201] The data processing device disclosed in this embodiment is implemented based on the data processing method disclosed in the above embodiment, and will not be elaborated here.

[0202] The data processing device disclosed in this embodiment obtains a target operation for a target image when the target image is output on the display screen of the electronic device; determines target information based on the target operation, where the target information can respond to the operation intention of the target operation; and outputs the target information. In this solution, when the target image is output on the display screen, if a target operation for the target image is obtained, the target operation is not directly executed. Instead, it is necessary to determine the operation intention of the target operation and determine the target information for responding to the operation intention, and then output the target information. Thus, when performing a target operation on the target image output on the display screen, the operation intention of the operation can be analyzed, and the target information for responding to the operation intention can be output, so as to ensure that the response to the operation intention of the target operation is directly output based on the target operation, without the user having to perform multiple operations or complex operations, simplifying the user operation and improving the user experience.

[0203] This embodiment discloses an electronic device, and its structural schematic diagram is as Figure 10 shown, including:

[0204] A display screen 101 and a processor 102.

[0205] Among them, the processor 102 is configured to obtain a target operation for the target image when the target image is output on the display screen; determine target information based on the target operation, where the target information can respond to the operation intention of the target operation; and output the target information through the display screen.

[0206] The electronic device disclosed in this embodiment is implemented based on the data processing method disclosed in the above embodiment, and will not be elaborated here.

[0207] The electronic device disclosed in this embodiment obtains a target operation for a target image when the target image is output on the display screen of the electronic device; determines target information based on the target operation, where the target information can respond to the operation intention of the target operation; and outputs the target information. In this solution, when the target image is output on the display screen, if a target operation for the target image is obtained, the target operation is not directly executed. Instead, it is necessary to determine the operation intention of the target operation and determine the target information for responding to the operation intention, and then output the target information. Thus, when performing a target operation on the target image output on the display screen, the operation intention of the operation can be analyzed, and the target information for responding to the operation intention can be output, so as to ensure that the response to the operation intention of the target operation is directly output based on the target operation, without the user having to perform multiple operations or complex operations, simplifying the user operation and improving the user experience.

[0208] In addition, it should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in the attached drawings of the device embodiments provided in this application, the connection relationships between the modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines.

[0209] Through the description of the above embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by dedicated hardware including application-specific integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. Generally, functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structures used to implement the same function can also be various, such as analog circuits, digital circuits or dedicated circuits. However, for this application, in more cases, software program implementation is a better implementation method. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disc of a computer, and includes several instructions to enable a computer device (which can be a personal computer, training device, or network device, etc.) to execute the methods described in various embodiments of this application.

[0210] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product.

[0211] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are all or partially generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that can be stored by a computer or a data storage device such as a training device or a data center that includes one or more integrated available media. The available medium may be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)).

Claims

1. A data processing method, comprising: When a target image is output on a display screen of an electronic device, obtaining a target operation for the target image; Determining target information based on the target operation, where the target information can respond to the operation intention of the target operation; Outputting the target information.

2. The method according to claim 1, wherein the determining target information based on the target operation comprises: In response to the target image having a first presentation mode, obtaining position information of the target operation on the target image, and determining output information corresponding to the position information on the target image, where the first presentation mode makes the image not have elements that can be operated independently; In response to the target image having a second presentation mode, obtaining output information displayed in response to the target operation, where the second presentation mode makes the image have at least one element that can be operated independently; Determining target information based on the output information.

3. The method according to claim 1, wherein the determining target information based on the target operation comprises: Analyzing the target image to determine the information included in the target image; Determining target information based on the target operation and the information included in the target image.

4. The method according to claim 3, wherein the determining target information based on the target operation and the information included in the target image comprises: If the information included in the information included in the target image includes a target object, querying information about the target object; Determining target information based on the target operation, the information included in the target image, and the information about the target object.

5. The method according to claim 1, if the target image is an image that can represent a data trend, the determining target information based on the target operation comprises: Analyzing the target image and the target operation for the target image to determine multiple fields and the data corresponding to each field respectively; Filling the multiple fields and the data corresponding to each field into a data structure corresponding to the target image, where at least two of the multiple fields included in the data structure include: the chart type corresponding to the data structure, the information about the operation point corresponding to the target operation, the keyword of the target image, the key information of the data structure; Determining target information based on the data structure.

6. The method according to claim 5, wherein the determining target information based on the data structure comprises: Inputting the data structure into a prompt word generation model; Obtaining at least one candidate question output by the prompt word generation model based on the data structure; In response to a selection operation on the at least one candidate question, inputting the target question selected based on the selection operation into a large model; Obtaining target information output by the large model based on the target question.

7. The method according to claim 1, wherein the determining target information based on the target operation comprises: Invoking a target model based on the target operation, and determining target information through the target model.

8. The method according to claim 7, the outputting the target information comprises at least one of the following: Output the target information through the output window of the target model; Output the target information at a target position on the display screen, where the target position of the display screen is at least related to the position where the display screen outputs the target image.

9. A data processing device, comprising: An obtaining unit, configured to obtain a target operation for the target image when a target image is output on a display screen of an electronic device; A determining unit, configured to determine target information based on the target operation, where the target information can respond to the operation intention of the target operation; An output unit, configured to output the target information.

10. An electronic device, comprising: A display screen; A processor, configured to obtain a target operation for the target image when the target image is output on the display screen; Determine target information based on the target operation, where the target information can respond to the operation intention of the target operation; Output the target information through the display screen.

Images