Text generation method and device, cloud server, electronic equipment and storage medium

Abstract

The invention discloses a text generation method and device, a cloud server, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence, the method is applied to the electronic equipment, and the method comprises the following steps: when it is detected that the electronic equipment displays a target interface, responding to a confirmation operation of a target user for a target control, a text recognition operation and an image recognition operation are executed on display content in the target interface to obtain to-be-analyzed data content, the target interface is used for receiving input content of a target user, and the to-be-analyzed data content comprises at least one of text content and image content; and performing semantic analysis on the to-be-analyzed data content through a large model algorithm to obtain target text information serving as the input content of the target user. And the convenience of obtaining the target text information by the target user through a large model algorithm is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a text generation method, device, cloud server, electronic device and storage medium. Background Art

[0002] At present, with the development of electronic information technology, users can use electronic devices to generate text corresponding to some content. However, currently, using electronic devices to generate text corresponding to content requires users to perform relatively cumbersome operations, which is less convenient. Summary of the invention

[0003] The present application proposes a text generation method, device, cloud server, electronic device and storage medium to improve the above-mentioned defects.

[0004] In a first aspect, an embodiment of the present application provides a text generation method, which is applied to an electronic device, and the method comprises: when it is detected that the electronic device displays a target interface, in response to a confirmation operation of a target user on a target control, performing a text recognition operation and an image recognition operation on the display content in the target interface to obtain data content to be analyzed, wherein the target interface is used to receive input content from a target user, and the data content to be analyzed includes at least one of text content and image content; performing semantic analysis on the data content to be analyzed by using a large model algorithm to obtain target text information as the input content of the target user.

[0005] In the second aspect, an embodiment of the present application also provides a text generation device, which is applied to an electronic device, and the device includes: an identification unit and a text generation unit. The identification unit is used to, when it is detected that the electronic device displays a target interface, respond to the target user's confirmation operation on the target control, perform text recognition operations and image recognition operations on the display content in the target interface to obtain data content to be analyzed, wherein the target interface is used to receive the input content of the target user, and the data content to be analyzed includes at least one of text content and image content; the text generation unit is used to perform semantic analysis on the data content to be analyzed through a large model algorithm to obtain target text information as the input content of the target user.

[0006] In a third aspect, an embodiment of the present application further provides a cloud server, which is used to obtain the data content to be analyzed obtained by the electronic device based on the method described in the first aspect, and perform semantic analysis on the data content to be analyzed through a large model algorithm to obtain target text information as the input content of the target user.

[0007] In a fourth aspect, an embodiment of the present application further provides an electronic device, comprising: one or more processors; a memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method described in the first aspect.

[0008] In a fifth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a program code executable by a processor, and when the program code is executed by the processor, the processor executes the above method.

[0009] The text generation method, device, cloud server, electronic device and storage medium provided by the present application, when it is detected that the electronic device displays a target interface, in response to the target user's confirmation operation on the target control, perform text recognition operation and image recognition operation on the display content in the target interface to obtain the data content to be analyzed, wherein the target interface is used to receive the input content of the target user, and the data content to be analyzed includes at least one of text content and image content; thereby, the data content to be analyzed is semantically analyzed by a large model algorithm to obtain the target text information as the input content of the target user. In the scheme of the present application, when it is detected that the electronic device displays a target interface, in response to the target user's confirmation operation input through the target control, the target text information corresponding to the display content in the target interface can be obtained, thereby improving the convenience for the target user to obtain the target text information.

[0010] Other features and advantages of the embodiments of the present application will be described in the subsequent description, and partly become apparent from the description, or can be understood by practicing the embodiments of the present application. The purposes and other advantages of the embodiments of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0012] Figure 1 A method flow chart of a text generation method provided by an embodiment of the present application is shown;

[0013] Figure 2 A schematic diagram of a target interface provided by an embodiment of the present application is shown;

[0014] Figure 3 A schematic diagram of a target interface provided by another embodiment of the present application is shown;

[0015] Figure 4 A schematic diagram showing target text information provided by an embodiment of the present application;

[0016] Figure 5 A method flow chart of a text generation method provided by another embodiment of the present application is shown;

[0017] Figure 6 A method flow chart of a text generation method provided by another embodiment of the present application is shown;

[0018] Figure 7 A method flow chart of a text generation method provided by another embodiment of the present application is shown;

[0019] Figure 8 A method flow chart of a text generation method provided by another embodiment of the present application is shown;

[0020] Fig. 9 Shows Figure 8 A flowchart of the implementation method of each step in;

[0021] Fig.10 A schematic diagram showing the interaction between the scene intelligence application and the screen recognition application provided in an embodiment of the present application is shown;

[0022] Fig.11 A flow chart showing the interaction between the scene intelligence application and the screen recognition application provided in an embodiment of the present application is shown;

[0023] Fig.12 A structural block diagram of obtaining the content of data to be analyzed provided in an embodiment of the present application is shown;

[0024] Fig.13 A structural block diagram of a text generation device provided in an embodiment of the present application is shown;

[0025] Fig.14 A schematic diagram of a cloud server provided in an embodiment of the present application is shown;

[0026] Fig.15 A structural block diagram of an electronic device provided in an embodiment of the present application is shown;

[0027] Fig.16 A structural block diagram of a computer-readable storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0028] In order to make those skilled in the art better understand the present application scheme, the technical scheme in the present application embodiment will be clearly and completely described below in conjunction with the drawings in the present application embodiment. Obviously, the described embodiment is only a part of the present application embodiment, rather than all the embodiments. The components of the present application embodiment usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the present application for protection, but merely represents the selected embodiment of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0029] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0030] At present, with the development of electronic information technology, users can use electronic devices to generate text corresponding to some content. However, currently, using electronic devices to generate text corresponding to content requires users to perform relatively cumbersome operations, which is less convenient. How to improve the convenience of generating text corresponding to content through electronic devices is a problem that needs to be solved urgently.

[0031] At present, in order to generate text corresponding to the content through an electronic device, the user can manually control the electronic device to run a designated application that generates text through artificial intelligence technology. Then, the content that needs to be analyzed by artificial intelligence technology is input into the designated application. For example, the content can be image content or text content. Therefore, after obtaining the content input by the user, the designated application can analyze the content based on artificial intelligence technology to obtain the text corresponding to the content.

[0032] However, the inventors discovered during their research that in the above-mentioned technical solution in which the user manually controls the electronic device to run a specified application to generate text corresponding to the content, the user is required to manually control the running of the specified application. In addition, the user is also required to input content into the specified application. The user needs to perform relatively complex and tedious operations, which is less convenient and reduces the user experience.

[0033] Therefore, in order to overcome the above-mentioned defects, the present application provides a text generation method, device, cloud server, electronic device and storage medium.

[0034] See also Figure 1 , Figure 1The text generation method provided by the embodiment of the present application is shown. The text generation method can be applied to an electronic device, and specifically, the processor of the electronic device can be used as the execution subject of the text generation method. The text generation method can include step S110 and step S120.

[0035] The text generation method provided in the embodiments of the present application can be applied to electronic devices, and exemplary electronic devices may be smart phones, smart tablet computers, laptop computers, servers, and the like.

[0036] Step S110: When it is detected that the electronic device displays a target interface, in response to a confirmation operation of a target user on a target control, a text recognition operation and an image recognition operation are performed on the display content in the target interface to obtain data content to be analyzed, wherein the target interface is used to receive input content from a target user, and the data content to be analyzed includes at least one of text content and image content.

[0037] The electronic device may run an operating system, so that the electronic device may also run various applications through the operating system to implement the functions corresponding to the application. The application may correspond to different display interfaces, and the target user may view the content displayed on the display interface. In some embodiments, some display interfaces may also be used to receive content input by the target user. For example, the display interface may include an input box, and the input box is used to receive the content input by the target user.

[0038] Therefore, in order to improve the convenience of the target user using the electronic device to generate text corresponding to the content, the display interface of the electronic device can be detected, and for the display interface including the display interface for receiving the input content of the target user, it can be further detected whether the confirmation operation of the target user for the target control is obtained. The user's input content can be text information.

[0039] Among them, the target control can be a control that the target user can interact with, so that the target user can input a confirmation operation through the target control. For example, the target control can be a first interactive control displayed in the target interface displayed by the electronic device; for another example, the target control can also be a physical button set on the electronic device, and specifically, for example, the target control can be a physical button set on the side of the electronic device object. Thus, the target user can interact with the displayed first interactive control, such as clicking the first interactive control; or interact with the physical button, such as pressing the physical button, to realize the input confirmation operation through the target control. For a detailed description, please refer to the subsequent embodiments.

[0040] For example, see Figure 2 , Figure 2A schematic diagram of a target interface provided by an embodiment of the present application is shown. Figure 2 In the target interface 200 shown in the figure, the target interface 200 is a sharing interface of a social application, and the target interface 200 includes an input box 210, in which the target user can input text information and then share the input text information.

[0041] For example, see Figure 3 , Figure 3 A schematic diagram of a target interface provided by an embodiment of the present application is shown. Figure 3 In the target interface 300 shown in FIG. 3 , the target interface 300 is a sharing interface of a community-type application. The target interface 300 includes an input box 310 , in which the target user can input text information and then share the input text information.

[0042] From the foregoing introduction, it can be seen that the target user can input a confirmation operation through the target control, so that the electronic device can respond to the target user's confirmation operation on the target control, perform text recognition operations and image recognition operations on the displayed content in the target interface, and obtain the data content to be analyzed.

[0043] The display content in the target interface may include image content and text content in the target interface.

[0044] That is, after detecting the target user's confirmation operation on the target control, the electronic device can automatically perform text recognition and image recognition operations on the display content in the target interface, thereby obtaining the data content to be analyzed. The data content to be analyzed may include at least one of text content and image content.

[0045] Step S120: semantically analyzing the data content to be analyzed by a large model algorithm to obtain target text information as input content of the target user.

[0046] Furthermore, after obtaining the data content to be analyzed, semantic analysis can be performed on the data content to be analyzed, thereby obtaining target text information corresponding to the data content to be analyzed, and the target text can be used as input content for the target user in the target interface.

[0047] In some embodiments, a large model algorithm may be called to perform semantic analysis on the data content to be analyzed. The large model algorithm may call a multimodal model so as to understand and process multimodal content (such as text content and image content, etc.) to generate text content corresponding to the multimodal content. The large model algorithm is an algorithm based on artificial intelligence technology.

[0048] Optionally, the large model algorithm can be deployed locally on the electronic device, so that after the electronic device obtains the data content to be analyzed, it can directly call the large model algorithm locally to perform semantic analysis on the data content to be analyzed and obtain target text information as the input content of the target user.

[0049] Optionally, the large model algorithm can also be deployed on a cloud server, and a communication connection is established between the electronic device and the cloud server. Thus, after the electronic device obtains the data content to be analyzed, it can upload the data content to be analyzed to the cloud server, so that the cloud server can call the large model algorithm to perform semantic analysis on the data content to be analyzed, and obtain the target text information as the input content of the target user. After obtaining the target text information, the cloud server can send the target text information to the electronic device through the communication connection. By deploying the large model algorithm on the cloud server, the storage space and computing resources of the electronic device are saved.

[0050] In some embodiments, the acquired target text information may be displayed in the target interface. For example, see Figure 4 , Figure 4 A schematic diagram of target text information provided by an embodiment of the present application is shown. Figure 4 The target interface 400 shown in FIG. 4 includes target text information 410, which is the target text information 410 obtained by the large model algorithm. Figure 2 The data to be analyzed is obtained by performing semantic analysis on the displayed content in the target interface 200 shown in FIG. 1 , by performing text recognition operations and image recognition operations on the obtained data content.

[0051] The text generation method provided by the present application, in the case where it is detected that the electronic device displays a target interface, in response to the target user's confirmation operation on the target control, performs a text recognition operation and an image recognition operation on the display content in the target interface to obtain the data content to be analyzed, wherein the target interface is used to receive the input content of the target user, and the data content to be analyzed includes at least one of text content and image content; thereby, the data content to be analyzed is semantically analyzed by a large model algorithm to obtain the target text information as the input content of the target user. In the scheme of the present application, in the case where it is detected that the electronic device displays a target interface, in response to the target user's confirmation operation input through the target control, the target text information corresponding to the display content in the target interface can be obtained, thereby improving the convenience for the target user to obtain the target text information.

[0052] See also Figure 5 , Figure 5The text generation method provided by an embodiment of the present application is shown. This text generation method can be applied to an electronic device, specifically, the processor of the electronic device can be used as the execution subject of this text generation method. This text generation method may include step S210 to step S230.

[0053] Step S210: When it is detected that the electronic device displays a target interface, a first interaction control is displayed at a specified position on the target interface.

[0054] In the text generation method provided by an embodiment of the present application, the target control can be the first interaction control. Thus, when it is detected that the electronic device displays a target interface, a first interaction control can be displayed at a specified position on the target interface. And the target interface is used to receive the input content of the target user.

[0055] Among them, the first interaction control can be a control with which the target user can perform interaction operations, so that the target user can input a confirmation operation through the first interaction control. Exemplarily, when the display screen configured by the electronic device is a touch screen, the target user can realize the interaction operation with the first interaction control through the touch screen. Specifically, for example, the target user can click on the first interaction control to input a confirmation operation through the first interaction control; or for another example, the target user can also click on the first interaction control and then slide, so as to input a confirmation operation through the first interaction control.

[0056] Optionally, the electronic device may also be configured with a voice input unit, such as a microphone, so that the target user can issue a voice to perform an interaction operation on the first interaction control. Specifically, for example, the target user can issue a voice of "click on the first interaction control", so that after the electronic device obtains the voice issued by the target user through the microphone, the voice can be recognized, and then it can be confirmed that the target user inputs a confirmation operation through the first interaction control.

[0057] It should be noted that the interaction operation of the target user on the first interaction control can be regarded as a confirmation operation.

[0058] Further, please continue to refer to Figure 2 , Figure 2 also shows the first interaction control 220. The first interaction control 220 is located above the display area of the virtual keyboard 230 in the target interface 200 and adjacent to the virtual keyboard 230, that is, the specified position is above the display area of the virtual keyboard 230 and adjacent to the virtual keyboard 230. Among them, Figure 2 the first interaction control 220 shown in is an icon with a text label, and the text label is "Ai Writing Assistant".

[0059] In addition, please continue to refer to Figure 3 , Figure 3 The first interactive control 320 is also shown. The first interactive control 320 is located above each interactive control in the target interface 300, that is, the designated position is the position above each interactive control. Figure 3 The first interactive control 320 shown in FIG. 1 is an icon with a text logo, and the text logo is “Ai Copywriting Assistant”.

[0060] It should be noted that the appearance, text identifiers and designated positions of the first interactive controls shown in the above schematic diagrams are examples and do not constitute specific limitations on the present application. In practical applications, they can be flexibly adjusted according to needs.

[0061] The display content in the target interface may include image content and text content in the target interface. Figure 2 , Figure 2 The target interface 200 includes image content 280 and text content 290 .

[0062] For some implementations, when it is detected that the electronic device displays the target interface, the display status of the virtual keyboard may be further obtained to determine whether to display the first interactive control. Specifically, step S210 may also include steps S211 to S214.

[0063] Step S211: when it is detected that the electronic device displays a target interface, obtaining a display status of a virtual keyboard in the target interface.

[0064] Step S212: Check whether the virtual keyboard is displayed.

[0065] When it is detected that the electronic device displays the target interface, the display status of the virtual keyboard in the target interface can be further obtained.

[0066] It is understandable that if the target user wants to input content on the target interface, he generally needs to input through a virtual keyboard. Therefore, if a displayed virtual keyboard is detected in the target interface, it can be determined that the target user has a high probability of needing to input content on the target interface, and thus, the first interactive control can be displayed at a specified position of the target interface.

[0067] That is to say, if it is detected that the virtual keyboard is displayed, the process can jump to step S213 to display the first interactive control at the specified position of the target interface; if it is detected that the virtual keyboard is not displayed, the process can jump to step S214 to not display the first interactive control.

[0068] Step S213: If it is detected that the virtual keyboard has been displayed, a first interactive control is displayed at a designated position of the target interface.

[0069] Optionally, when the first interactive control is displayed at the designated position of the target interface, it is also possible to continuously monitor whether the virtual keyboard is displayed. If it is detected that the virtual keyboard is switched from being displayed to not being displayed, the first interactive control can be hidden.

[0070] In some implementations, the designated position may include a position above the display area of ​​the virtual keyboard and adjacent to the virtual keyboard. Figure 2 , Figure 2 The position of the first interactive control 220 shown is above the display area of ​​the virtual keyboard 230 in the target interface 200 and adjacent to the virtual keyboard 230, that is, the designated position.

[0071] Step S214: If it is detected that the virtual keyboard is not displayed, the first interactive control may not be displayed.

[0072] Among them, not displaying the first interactive control can be understood as hiding the first interactive control.

[0073] Step S220: In response to the target user's confirmation operation on the first interactive control, a text recognition operation and an image recognition operation are performed on the display content in the target interface to obtain data content to be analyzed.

[0074] After detecting the target user's confirmation operation on the first interactive control, the electronic device can automatically perform text recognition and image recognition operations on the display content in the target interface to obtain the data content to be analyzed. The data content to be analyzed may include at least one of text content and image content. Figure 2 Taking the target interface 200 shown in the figure as an example, a text recognition operation and an image recognition operation are performed on the display content in the target interface 200 to obtain data content to be analyzed, namely, text content and image content.

[0075] Step S230: semantically analyzing the data content to be analyzed by a large model algorithm to obtain target text information as input content of the target user.

[0076] In the text generation method provided in the embodiment of the present application, it can be determined whether to display the first interactive control according to the display of the virtual keyboard. Thus, when it is detected that the virtual keyboard has been displayed, the first interactive control is displayed at the specified position of the target interface. The target user can thus directly perform a confirmation operation through the first interactive control to trigger a subsequent response to the target user's confirmation operation on the first interactive control, perform a text recognition operation and an image recognition operation on the display content in the target interface, obtain the data content to be analyzed, and then perform a semantic analysis on the data content to be analyzed through a large model algorithm to obtain the target text information as the input content of the target user. In other words, the target user no longer needs to actively search for or control the startup of the relevant application for obtaining the data content to be analyzed through artificial skill technology, and then manually input the display content in the target interface into the relevant application, which greatly improves the convenience of the user in obtaining the target text information.

[0077] See also Figure 6 , Figure 6 The text generation method provided by the embodiment of the present application is shown. The text generation method can be applied to an electronic device, and specifically, the processor of the electronic device can be used as the execution subject of the text generation method. The text generation method can include steps S310 to S350.

[0078] Step S310: When it is detected that the electronic device displays a target interface, in response to a confirmation operation of the target user on a target control, a text recognition operation and an image recognition operation are performed on the display content in the target interface to obtain data content to be analyzed, wherein the target interface is used to receive input content from the target user, and the data content to be analyzed includes at least one of text content and image content.

[0079] Step S320: semantically analyzing the data content to be analyzed by a large model algorithm to obtain target text information as input content of the target user.

[0080] Among them, step S310 and step S320 have been introduced in detail in the above embodiments and will not be repeated here.

[0081] Step S330: Display the second interactive control.

[0082] Step S340: In response to the target user's confirmation operation on the second interactive control, the target text information is filled into an input box in the target interface for receiving input content of the target user.

[0083] In some embodiments, after obtaining the target text information, a second interactive control may be displayed. Similar to the first interactive control, the second interactive control is also a control that the target user can interact with, so that the target user can input a confirmation operation through the second interactive control. The introduction of the target user inputting a confirmation operation through the second interactive control is similar to the aforementioned target user inputting a confirmation operation through the first interactive control, and will not be repeated here.

[0084] Thus, when the electronic device detects the target user's confirmation operation on the second interactive control, it can fill the target text information into the input box in the target interface for receiving the target user's input content in response to the target user's confirmation operation on the second interactive control.

[0085] That is to say, in the text generation method provided in the embodiment of the present application, after obtaining the target text information, a second interactive control can also be displayed, so that when a confirmation operation of the target user on the second interactive control is detected, the target text information can be automatically filled into the input box in the target interface for receiving the input content of the target user, so as to further improve the convenience for the target user to obtain the target text information using artificial intelligence technology.

[0086] For example, please refer to Figure 4 , Figure 4 The target text information 410 is shown in Figure 4 The second interactive control 420 is also shown. Thus, the target user can perform a confirmation operation through the second interactive control 420 to fill the target text information 410 into Figure 2 In the input box 210 of the target interface 200.

[0087] It should be noted that filling the target text information into the input box is essentially that the electronic device copies the target text information and pastes it into the input box. In other words, the electronic device automatically completes the operation of inputting the same text information as the target text information into the input box.

[0088] In some embodiments, the electronic device can find the text editing (EditText) control corresponding to the target interface by traversing the target interface. Exemplarily, the operating system running the electronic device can be configured with an accessibility function, so that the ViewTree node of the target interface can be traversed to determine whether the target interface corresponds to a text editing control. In the case of determining that the target interface corresponds to a text editing control, it can be determined that the text editing control corresponds to an input box for obtaining user input content. Thus, the data interface provided by the accessibility function can be directly called to fill the target text information obtained through the aforementioned steps into the input box.

[0089] Optionally, when it is determined that the target interface corresponds to multiple text editing controls, the target text information may be filled into an input box corresponding to the last detected text editing control.

[0090] Step S350: If the target text information is not successfully filled into the input box in the target interface for receiving the input content of the target user, the target text information is stored and prompt information is generated.

[0091] In addition, even if the target user's confirmation operation on the second interactive control is obtained, the electronic device may not be able to successfully fill the target text information into the input box in the target interface for receiving the input content of the target user. Therefore, if the target text information is not successfully filled into the input box in the target interface for receiving the input content of the target user, the electronic device can store the target text information and generate prompt information.

[0092] Exemplarily, the electronic device fails to find a text editing control by traversing the ViewTree nodes of the target interface, and thus can determine that the target text information is not successfully filled into the input box in the target interface for receiving the input content of the target user.

[0093] Exemplarily, if the electronic device fails to start the accessibility function, for example, the startup times out for a specified period of time, such as 3 seconds, then it can be determined that the target text information is not successfully filled into the input box in the target interface for receiving the input content of the target user.

[0094] Among them, the electronic device can be configured with a storage unit, so that the target text information can be stored in the storage unit. Optionally, the storage unit can generally be divided into volatile storage and non-volatile storage. For example, the electronic device can store the target text information in the clipboard of the operating system, and the clipboard corresponds to the storage area in the volatile storage. For another example, the electronic device can also store the target text information in the area storage area corresponding to the non-volatile storage.

[0095] In addition, a prompt message may be generated. The prompt message may be used to instruct the target user to manually paste the target text information into the input box of the target interface. Since the target text information has been stored, after seeing the prompt message, the user may directly control the electronic device to paste the target text information into the input box of the target interface without having to first store the target text information in the clipboard, which greatly improves convenience.

[0096] The text generation method provided by the present application can also display a second interactive control after performing semantic analysis on the data content to be analyzed by a large model algorithm to obtain the target text information as the input content of the target user; in response to the target user's confirmation operation on the second interactive control, the target text information is filled into the input box in the target interface for receiving the input content of the target user. In other words, the target text information can be automatically filled into the input box in the target interface for receiving the input content of the target user, without the need for the target user to manually input the target text information into the input box in the target interface based on the acquired target text information, thereby reducing the target user's operation steps and learning costs, and further improving the convenience of the target user using artificial intelligence technology to obtain the target text information.

[0097] See also Figure 7 , Figure 7 The text generation method provided by the embodiment of the present application is shown. The text generation method can be applied to an electronic device, and specifically, the processor of the electronic device can be used as the execution subject of the text generation method. The text generation method can include steps S410 to S470.

[0098] Step S410: Detect whether the interface displayed by the electronic device matches a preset reference interface.

[0099] Step S420: When the interface displayed by the electronic device matches the reference interface, it is determined that the electronic device is detecting displaying a target interface.

[0100] In some embodiments, a reference interface may be preset, for example, the reference interface may be preset by an application developer. The reference interface may include multiple interfaces. It is understood that the reference interface is a preset interface that needs to be monitored.

[0101] It should be noted that the reference interface is the display interface of an application or mini-program. In the embodiments provided in the present application, the application corresponding to the reference interface is not limited to the first-party application of the electronic device, but can be used in suitable third-party applications in actual applications, which greatly improves the adaptability flexibility of the text generation method provided in the embodiments of the present application.

[0102] Thus, the interface displayed by the electronic device can be detected, and when the interface displayed by the electronic device matches the reference interface, it is determined that the electronic device is detecting displaying the target interface.

[0103] In some implementations, the detection of the interface displayed by the electronic device can be implemented through a scene intelligent service, wherein the scene intelligent service can be provided by a scene intelligent application program or by an operating system, which is not specifically limited in the embodiments of the present application.

[0104] Optionally, the scene intelligent service can monitor the displayed interface through a specified service in the operating system. For example, the specified service may be the Op l usAppSwitchManager service.

[0105] Optionally, the scene intelligent service can obtain a preset reference interface, so as to facilitate the subsequent detection of whether the interface displayed by the electronic device matches the preset reference interface. The scene intelligent service can obtain each reference interface through the SceneServiceProvider interface.

[0106] When the scene intelligent service detects that the interface displayed by the electronic device matches the reference interface, it can be determined that the electronic device is detected to display the target interface. Thus, the screen recognition application can be awakened to perform subsequent steps through the screen recognition application to obtain the data content to be analyzed. Exemplarily, the scene intelligent service can wake up the screen recognition application through the scene entry service (SceneEntryService).

[0107] Optionally, a reference text may be preset, and then the text in the interface displayed by the electronic device may be obtained to determine whether the text contains the reference text. Thus, if the reference text is contained, it may be determined that the electronic device displays the target interface.

[0108] It is understandable that the determined reference interface may change in different scenarios. Therefore, in some implementations, step S410 may further include steps S411 to S413.

[0109] Step S411: Identify scene information corresponding to the interface displayed by the electronic device.

[0110] Step S412: Determine a reference interface corresponding to the scene information based on pre-set scene interface reference data, wherein the scene interface reference data includes at least one standard scene information and a standard interface corresponding to each standard scene information.

[0111] First, the scene information corresponding to the interface displayed by the electronic device can be identified. For example, if the displayed interface is a chat interface of an instant messaging software, the corresponding scene information can be determined to be a chat scene; if the displayed interface is a sharing interface of a social software, the corresponding scene information can be determined to be a sharing scene.

[0112] In some implementations, the interface displayed by the electronic device may be analyzed using a pre-trained model to determine the corresponding scene information. For example, the pre-trained model may be a neural network model.

[0113] Thus, after obtaining the scene information, the reference interface corresponding to the scene information can be further determined. Specifically, the reference interface corresponding to the scene information can be determined based on pre-set scene interface reference data. The scene interface reference data includes at least one standard scene information and a standard interface corresponding to each standard scene information.

[0114] Exemplarily, the scene interface comparison data may be a scene interface comparison table. Thus, a standard scene information matching the scene information may be searched in the scene interface comparison data, and then the standard interface corresponding to the standard scene information matching the scene information may be used as the reference interface corresponding to the interface displayed by the electronic device. Thus, a subsequent detection may be performed to determine whether the interface displayed by the electronic device matches the reference interface. In some embodiments, after acquiring the scene interface comparison data, the target user may also update the scene interface comparison data. Specifically, step S412 may also include steps S413 to S416.

[0115] Step S413: Acquire scene interface comparison data.

[0116] Step S414: in response to the target user's update operation on the scene interface reference data, obtaining the designated scene information determined by the target user and the designated interface corresponding to the designated scene information.

[0117] Step S415: The designated scene information and the designated interface corresponding to the designated scene information are added to the scene interface comparison data as new standard scene information and a new standard interface matching the new standard scene information, respectively, to update the scene interface comparison data.

[0118] Step S416: Determine a reference interface that matches the scene information based on the updated scene interface comparison data.

[0119] First, the scene interface comparison data can be obtained. It is understandable that the target user may need to add another interface as the standard interface corresponding to the scene information according to his own needs when in some scene information. Alternatively, the user may also need to add some new scene information and the standard interface corresponding to the new scene information.

[0120] Therefore, the user can update the scene interface reference data, for example, the user can update the scene interface reference data through the electronic device. Thus, the electronic device can obtain the specified scene information determined by the target user and the specified interface corresponding to the specified scene information in response to the target user's update operation on the scene interface reference data.

[0121] It is understandable that the target user can determine the designated scene information and the designated interface corresponding to the designated scene information through the electronic device.

[0122] Exemplarily, the target user may select standard scene information existing in the scene interface comparison data as the designated scene information, and then set a corresponding designated interface for the designated scene information.

[0123] Exemplarily, the target user may also set new scene information other than the standard scene information already existing in the scene interface comparison data as designated scene information, and then set a corresponding designated interface for the designated scene information.

[0124] Furthermore, the designated scene information and the designated interface corresponding to the designated scene information may be respectively added to the scene interface comparison data as new standard scene information and a new standard interface matching the new standard scene information to update the scene interface comparison data.

[0125] It should be noted that when updating the scene interface comparison data, if there is already standard scene information in the scene interface comparison data that is consistent with the new standard scene information, the new standard interface matching the new standard scene information can be directly added to the scene interface comparison data, and the standard interface originally corresponding to the consistent standard scene information can be used as the standard interface corresponding to the consistent standard scene information.

[0126] If there is no standard scene information consistent with the new standard scene information in the scene interface comparison data, the new standard scene information and the new standard interface matching the new standard scene information can be directly added to the scene interface comparison data to update the scene interface comparison data.

[0127] Then, based on the updated scene interface comparison data, a reference interface matching the scene information may be determined.

[0128] In some embodiments, the scene interface control data may be included in the configuration information, so that the subsequent scene intelligent application can obtain the configuration information and then obtain the scene interface control data. For example, the screen recognition application may include the configuration information, so that the configuration information may be sent to the scene intelligent application through the screen recognition application. In addition, the target user may also update the configuration information through the screen recognition application, specifically, update the scene interface control data in the configuration information.

[0129] Step S413: Detect whether the interface displayed by the electronic device matches the reference interface.

[0130] Then, it is detected whether the interface displayed by the electronic device matches the reference interface.

[0131] Thus, the target user can determine the designated scene information and the designated interface corresponding to the designated scene information according to his own needs, and then add the designated scene information and the designated interface corresponding to the designated scene information to the photo data to update the scene interface control data. When the reference interface corresponding to the interface displayed by the electronic device is subsequently determined by the scene interface control data, the reference interface can include the reference interface pre-added by the target user according to his own needs, achieving a certain degree of customization of the reference interface.

[0132] Step S430: When it is detected that the electronic device displays the target interface, in response to the target user's confirmation operation on the target control, a text recognition operation and an image recognition operation are performed on the display content in the target interface to obtain data content to be analyzed.

[0133] After obtaining the target user's confirmation operation for the target control, a text recognition operation and an image recognition operation can be performed on the display content in the target interface to obtain the data content to be analyzed. In some embodiments, the screen recognition application may also include a content extraction service, so that the text recognition operation and the image recognition operation can be performed on the display content in the target interface through the content extraction service.

[0134] See also Fig.12 , Fig.12 The following is a structural diagram of obtaining the content of the data to be analyzed provided by an embodiment of the present application. Fig.1211 shows a screen recognition application 1110, which includes an entry service 1111, an extraction service 1112, and an artificial intelligence business function module 1113. Specifically, after the screen recognition application obtains the target user's confirmation operation on the target control through the entry service 1111, the extraction service 1112 can perform text recognition and image recognition operations on the display content in the target interface to obtain the data content to be analyzed.

[0135] The extraction service 1112 can perform text recognition and image recognition operations on the display content in the target interface through the accessibility service 11121, Webview 11122, the image and text extraction framework 11123 corresponding to the H5 page, the image and text extraction framework 11124 corresponding to the Nat ive page, the screenshot service 11125 and the optical character recognition 11126 to obtain the data content to be analyzed. For a detailed introduction, please refer to the subsequent introduction.

[0136] Then, the data content to be analyzed is sent to the AIun it algorithm interface 1120 through the AIun it algorithm interface 1120, and the data content to be analyzed is uploaded to the cloud server 1130 through the AIun it algorithm interface 1120, and the target text information corresponding to the data content to be analyzed is obtained through the cloud server 1130. In addition, the screen recognition application 1110 can also call the system framework service 1140 in the process of obtaining the data content to be analyzed by performing text recognition and image recognition operations on the display content in the target interface through the extraction service 1112, wherein the system framework service 1140 includes barrier-free service 1141, image and text extraction framework service 1142 and screenshot service 1143, etc.

[0137] In some implementations, text recognition and image recognition operations can be performed on the display content in the target interface based on the accessibility service to obtain the data content to be analyzed. Fig.12 The accessibility service 11121 shown in the figure is used to obtain the data content to be analyzed.

[0138] In some implementations, the data content to be analyzed may also be obtained through a Uniform Resource Locator (URL). Specifically, step S430 may include steps S4301 to S4305.

[0139] Step S4301: when it is detected that the electronic device displays a target interface, in response to a confirmation operation of a target user on a target control, obtaining a first uniform resource locator corresponding to the target interface.

[0140] Step S4302: Obtain a hypertext markup language for describing the target page based on the first uniform resource locator.

[0141] Step S4303: extracting a second uniform resource locator corresponding to a first identifier from the hypertext markup language, wherein the first identifier is used as identity information of a first image, and the first image includes image content in the display content.

[0142] Step S4304: Acquire a first image based on the second uniform resource locator.

[0143] Step S4305: extracting the text content displayed in the target page from the hypertext markup language as the first text content, and taking the first image and the first text content as the data content to be analyzed.

[0144] When the target interface is a page in a browser application, when it is detected that the electronic device displays the target interface, in response to a confirmation operation of the target user on the target control, the first uniform resource locator corresponding to the target interface is first obtained.

[0145] Exemplarily, the first uniform resource locator of the browser target interface can be extracted through the image and text extraction framework. Specifically, for example, the first uniform resource locator can be obtained by searching for a text editor in the ViewTree.

[0146] Then, a hypertext markup language for describing the target page can be obtained based on the first uniform resource locator. Specifically, the web page content corresponding to the first uniform resource locator can be loaded through WebView, and then the hypertext markup language (HyperText Markup Language, HTML) of the target page can be obtained from the web page content. Specifically, after Webview has completed loading the web page content (onPageFin i sh is detected), the hypertext markup language of the target page can be obtained from the web page content.

[0147] Optionally, the WebView may be set to be transparent so that the process of loading the webpage content corresponding to the first uniform resource locator is invisible to the target user, thereby improving the user experience.

[0148] Furthermore, after acquiring the hypertext markup language, a second uniform resource locator corresponding to the first identifier may be extracted from the hypertext markup language, wherein the first identifier serves as identity information of the first image, and the first image includes image content in the display content.

[0149] For example, you can filter the hypertext markup language to get The tag is used to represent the first identifier. In addition, the tag also includes a second uniform resource locator. Therefore, the first image can be obtained based on the second uniform resource locator.

[0150] Then, the text content of the target page is extracted from the hypertext markup language as the first text content, and the first image and the first text content are used as the data content to be analyzed. Thus, the data content to be analyzed corresponding to the display content of the target interface is obtained. Fig.12 The Webview11122 shown in the figure is used to obtain the data content to be analyzed.

[0151] For some implementations, the data content to be analyzed may also be directly obtained through the image and text extraction framework. Specifically, step S430 may also include step S4306 and step S4307.

[0152] Step S4306: When it is detected that the electronic device displays the target interface, in response to the target user's confirmation operation on the target control, the image content of the content displayed in the target interface is obtained as the second image through the image and text extraction framework, and the text content of the content displayed in the target interface is obtained as the second text content.

[0153] Step S4307: taking the second image and the second text content as the data content to be analyzed.

[0154] When the target interface is a page in a non-browser application, when it is detected that the electronic device displays the target interface, in response to the target user's confirmation operation on the target control, the image content and text content displayed in the target interface can be obtained through the image and text extraction framework.

[0155] Exemplarily, a non-browser application may include an H5 page or a Nat ive page, wherein an H5 page is a page generated in the form of a web page design based on HTML5 technology, and a Nat ive page is a native application page, which is a page constructed based on the development language and framework of a specific platform (such as an Android platform, an iOS platform, etc.). Thus, in the case where the page in the non-browser application is an H5 page, the image content of the content displayed in the target interface can be obtained as the second image through the image-text extraction framework corresponding to the H5 page, and the text content of the content displayed in the target interface can be obtained as the second text content. In the case where the page in the non-browser application is a Nat ive page, the image content of the content displayed in the target interface can be obtained as the second image through the image-text extraction framework corresponding to the Nat ive page, and the text content of the content displayed in the target interface can be obtained as the second text content.

[0156] Furthermore, the second image and the second text content are used as the data content to be analyzed. Thus, the data content to be analyzed corresponding to the display content of the target interface is obtained. Fig.12 The image and text extraction framework 11123 corresponding to the H5 page and the image and text extraction framework 11124 corresponding to the Natural page shown in the figure are used to obtain the data content to be analyzed.

[0157] For some implementations, the data content to be analyzed corresponding to the display content of the target interface may also be obtained by taking a screenshot. Specifically, step S430 may also include step S4308 and step S4309.

[0158] Step S4308: When it is detected that the electronic device displays the target interface, in response to a confirmation operation of the target user on the target control, a screenshot operation is performed on the display content in the target interface to obtain a third image.

[0159] Step S4309: taking the third image as the data content to be analyzed.

[0160] When the target interface is a page of a specific application, when it is detected that the electronic device displays the target interface, in response to the target user's confirmation operation on the target control, a screenshot operation can be directly performed on the display content in the target interface to obtain a third image. Thus, the third image is used as the data content to be analyzed to obtain the data content to be analyzed. Fig.12 The screenshot service 11125 shown in the figure is used to obtain the data content to be analyzed.

[0161] The screenshot operation may be implemented by a built-in screenshot function in the operating system running on the electronic device, and the screenshot operation may generate an image file including the displayed content based on the displayed content in the target interface.

[0162] The specific application may be an application pre-selected by the user.

[0163] For some implementations, optical character recognition (OCR) may be used to obtain the data content to be analyzed corresponding to the display content of the target interface. Specifically, step S430 may also include steps S43010 to S4312.

[0164] Step S4310: When it is detected that the electronic device displays a target interface, in response to a confirmation operation of a target user on a target control, text recognition is performed on display content in the target interface through optical character recognition to obtain third text content.

[0165] Step S4311: taking a screenshot of the image content in the display content in the target interface to generate a fourth image.

[0166] Step S4312: taking the third text content and the fourth image as the data content to be analyzed.

[0167] In the case where the target interface belongs to a page of a mini program, when it is detected that the electronic device displays the target interface, in response to the target user's confirmation operation on the target control, text recognition is performed on the display content in the target interface through optical character recognition to obtain a third text content. In addition, a screenshot operation is performed on the image content in the display content in the target interface to generate a fourth image. The screenshot operation on the image content in the display content in the target interface can be implemented by a regional screenshot.

[0168] Thus, the third text content and the fourth image are used as the data content to be analyzed. Fig.12 The optical character recognition 11126 shown in the figure is used to obtain the data content to be analyzed.

[0169] Among them, mini programs are lightweight applications that do not need to be downloaded and installed like traditional mobile applications. Users can run mini programs directly in other applications with mini program interface functions, which greatly saves users' time and storage space.

[0170] Step S440: semantically analyzing the data content to be analyzed by a large model algorithm to obtain target text information as input content of the target user.

[0171] Among them, step S440 has been introduced in detail in the above embodiment and will not be repeated here.

[0172] In the text generation method provided in the embodiment of the present application, it can be detected whether the interface displayed by the electronic device matches the preset reference interface; when the interface displayed by the electronic device matches the reference interface, it is determined that the target interface is detected by the electronic device; when the target interface is detected, in response to the target user's confirmation operation on the target control, the display content in the target interface is subjected to text recognition and image recognition operations to obtain the data content to be analyzed. That is, the target user can determine the specified scene information and the specified interface corresponding to the specified scene information according to his own needs, and then add the specified scene information and the specified interface corresponding to the specified scene information to the reference data to update the scene interface control data. When the reference interface corresponding to the interface displayed by the electronic device determined by the scene interface control data is subsequently determined, the reference interface can include the reference interface pre-added by the target user according to his own needs, achieving a certain degree of customization of the reference interface.

[0173] See also Figure 8 , Figure 8 The text generation method provided by the embodiment of the present application is shown. The text generation method can be applied to an electronic device, and specifically, the processor of the electronic device can be used as the execution subject of the text generation method. The text generation method can include steps S510 to S550.

[0174] Step S510: The scene intelligence application detects whether the interface displayed by the electronic device matches a preset reference interface.

[0175] Among them, a scene intelligence application can be used to detect whether the interface displayed by the electronic device matches a pre-set reference interface.

[0176] For details, please refer to Fig. 9 , Fig. 9 Shows Figure 8 Flow chart of the implementation method of each step in . Figure 8 Step S510 shown in FIG. 1 may include Fig. 9 Step S511 and step S512 in .

[0177] Step S511: Detect whether the interface displayed by the electronic device matches a preset reference interface.

[0178] Step S512: Wake up the screen recognition application

[0179] Specifically, the scene intelligence application can wake up the screen recognition application when the interface displayed by the electronic device matches the reference interface.

[0180] Step S520: The screen recognition application calls the content extraction service to obtain the data content to be analyzed.

[0181] Among them, in the process of obtaining the data content to be analyzed, the screen recognition application can also call the system framework service, where the system framework service includes accessibility service, image and text extraction framework service and screenshot service.

[0182] Figure 8 Step S520 shown in FIG. 1 may include Fig. 9 Steps S521 to S526 in .

[0183] Step S521: Start the screen recognition application.

[0184] Step S522: Detect whether the virtual keyboard is displayed.

[0185] Step S523: If it is detected that the virtual keyboard has been displayed, a first interactive control is displayed at a designated position of the target interface.

[0186] Step S524: The target user performs a confirmation operation on the first interactive control.

[0187] Step S525: Obtain the data content to be analyzed.

[0188] Optionally, the appearance of the first interactive control may be a bubble shape.

[0189] In response to the target user's confirmation operation on the first interactive control, a text recognition operation and an image recognition operation can be performed on the display content in the target interface to obtain the data content to be analyzed. After obtaining the data content to be analyzed, the first data can be uploaded to the cloud server through the algorithm interface.

[0190] Also, see Fig.10 , Fig.10 A schematic diagram of the interaction between the scene intelligent application and the screen recognition application provided in an embodiment of the present application is shown. Fig.10 It includes a scene intelligence application 910 and a screen recognition application 920.

[0191] Among them, the scene intelligence application 910 can monitor the reference interface, so as to wake up the screen recognition application 920 when the interface displayed by the electronic device matches the reference interface.

[0192] The screen recognition application 920 may include a configuration query unit 921, a configuration manager 922, and a configuration data update unit 923. The configuration query unit 921 may obtain configuration information from the configuration manager 922 and the configuration data update unit 923 through the scene service provider (SceneServiceProvider) interface. The configuration information includes a preset reference interface. The screen recognition application 920 also includes a database 926. Specifically, the configuration information may be obtained by querying the database 926 through the configuration manager 922 and the configuration data update unit 923. The configuration manager 922 corresponds to the scene rule manager (SceneRu leManager) interface; the configuration data update unit 923 corresponds to the scene rule data source (SceneRu leDataSource) interface; and the database 926 corresponds to the SceneRuleEntityDao interface.

[0193] The screen recognition application 920 may also include a configuration synchronization unit 924, so that the configuration information is sent to the scene intelligence application 910 through the synchronization unit 924 to update the reference interface pre-acquired in the scene intelligence application 910. The configuration information may include scene interface reference data, so that the scene intelligence application 910 can determine the reference interface matching the scene information according to the scene interface reference data after acquiring the configuration information. The configuration synchronization unit 924 corresponds to the scene service manager (SceneServiceManager) interface.

[0194] In addition, the screen recognition application 920 may further include a monitoring unit 925. The monitoring unit 925 may monitor the display of the virtual keyboard through the SceneEntryService interface, so that when it is detected that the virtual keyboard has been displayed, the first interactive control is displayed at a specified position of the target interface.

[0195] Please also refer to Fig.10 as well as Fig.11 ,in, Fig.11 A flow chart showing the interaction between the scene intelligence application and the screen recognition application provided in an embodiment of the present application is shown.

[0196] in, Fig.11 The steps corresponding to the scene intelligent application and the steps corresponding to the screen recognition application are shown in FIG. Fig.11 It can be seen that the scene intelligent application can trigger the awakening of the screen recognition application, thereby triggering the start of the screen recognition application. Optionally, after the screen recognition application is started, the configuration information can be updated through the configuration synchronization unit.

[0197] The scene intelligence application can also query the screen recognition application for configuration information, where the configuration information includes multiple reference interfaces. After obtaining the query configuration information, the screen recognition application can return the updated configuration information to the scene intelligence application through the scene rule manager and the scene rule data source. Specifically, the database can be queried through the scene rule data source. If the data in the database is empty, the pre-set default configuration information can be returned to the scene intelligence application; if the data in the database is not empty, the data update can be triggered, and the updated configuration information can be sent to the scene intelligence application.

[0198] In addition, when the scene intelligence application detects that the electronic device displays the target interface, it can determine that the interface has been entered, so that the screen recognition application triggers the monitoring service to monitor the display of the virtual keyboard. Optionally, the screen recognition application can also obtain relevant data of each reference interface from the scene rule manager to more accurately monitor the display of the virtual keyboard in the target interface. Thus, when it is detected that the virtual keyboard has been displayed, the first interactive control can be displayed at the specified position of the target interface. In some embodiments, the first interactive control can also be referred to as a bubble service. Among them, the scene intelligence application can call Bindservice to the screen recognition application so that the screen recognition application triggers the monitoring service.

[0199] Furthermore, the scene intelligent application can also monitor whether the reference interface is exited. For example, if the reference interface is switched to the interface of other applications, it can be determined that the reference interface is exited. At this time, the monitoring service can be exited, specifically including canceling the monitoring of the display of the virtual keyboard. In addition, the first interactive control can also be hidden. Among them, the scene intelligent application can call UnBindservice to the screen recognition application so that the screen recognition application triggers the exit of the monitoring service.

[0200] Optionally, the scene intelligence application may not call UnBindservice to the screen recognition application normally, causing the screen recognition application to continue to monitor the display of the virtual keyboard. Therefore, in the screen recognition application, a timer task can also be set. In the timer task, the monitoring of the display of the virtual keyboard and the monitoring of whether the reference interface is exited can be canceled after a specified time, and the related processes of the screen recognition application can be canceled. Exemplarily, the specified time can be 2 hours.

[0201] In some implementations, the screen recognition application may be Ai Master.

[0202] Step S530: Upload the data content to be analyzed to the cloud server through the algorithm interface.

[0203] Among them, the algorithm interface can be provided by Aiun it.

[0204] Optional, Figure 8 Step S530 shown in FIG. 1 may include Fig. 9 Step S531 in .

[0205] Step S531: The data to be analyzed is pre-processed and uploaded to the cloud server.

[0206] Optionally, after obtaining the data content to be analyzed, Aiun it can pre-process the image content in the data content to be analyzed, for example, the pre-processing includes image enhancement, so as to upload the pre-processed data content to be analyzed to the cloud server.

[0207] Step S540: The cloud server calls the large model algorithm to perform semantic analysis on the data content to be analyzed, and obtains target text information as the input content of the target user.

[0208] The cloud server can call the large model algorithm to perform semantic analysis on the data content to be analyzed, and obtain the target text information as the input content of the target user.

[0209] Furthermore, Figure 8 Step S530 shown in FIG. 1 may also include Fig. 9 Step S532 in .

[0210] Step S532: Return the target text information to the screen recognition application.

[0211] Specifically, the cloud server can return the target text information to the screen recognition application through the algorithm interface.

[0212] Furthermore, Figure 8 Step S520 shown in FIG. 1 may also include Fig. 9 Steps S526 to S528 in .

[0213] Step S526: Display the target text information.

[0214] Step S527: Display the second interactive control.

[0215] Step S528: Fill the target text information into the input box.

[0216] In response to the target user's confirmation operation on the second interactive control, the target text information is filled into an input box in the target interface for receiving input content of the target user.

[0217] After acquiring the target text information, the electronic device can display it on the display screen of the electronic device through the screen recognition application. For example, the target text information can be displayed in a layer superimposed on the target interface.

[0218] The detailed description of each of the above steps can be found in the aforementioned embodiments and will not be repeated here.

[0219] See also Fig.13 , which shows a structural block diagram of a text generation device provided in an embodiment of the present application. The text generation device 1300 includes: a recognition unit 1310 and a text generation unit 1320.

[0220] The identification unit 1310 is used to, when it is detected that the electronic device displays a target interface, perform text recognition operations and image recognition operations on the display content in the target interface in response to a confirmation operation of the target user on the target control, so as to obtain data content to be analyzed, wherein the target interface is used to receive input content of the target user, and the data content to be analyzed includes at least one of text content and image content.

[0221] Optionally, the identification unit 1310 can also be used to display a first interactive control at a specified position of the target interface when it is detected that the electronic device displays the target interface; in response to the target user's confirmation operation on the first interactive control, perform text recognition operations and image recognition operations on the display content in the target interface to obtain data content to be analyzed.

[0222] Optionally, the identification unit 1310 can also be used to obtain the display status of the virtual keyboard in the target interface when it is detected that the electronic device displays the target interface; if it is detected that the virtual keyboard is displayed, display the first interactive control at a specified position of the target interface. The specified position includes a position above the display area of ​​the virtual keyboard and adjacent to the virtual keyboard.

[0223] Optionally, the identification unit 1310 can also be used to, when it is detected that the electronic device displays a target interface, obtain a first uniform resource locator corresponding to the target interface in response to a confirmation operation of a target user on a target control; obtain a hypertext markup language for describing the target page based on the first uniform resource locator; extract a second uniform resource locator corresponding to the first identifier from the hypertext markup language, wherein the first identifier serves as identity information of a first image, and the first image includes image content in the displayed content; obtain the first image based on the second uniform resource locator; extract the text content of the displayed content in the target page from the hypertext markup language as the first text content, and use the first image and the first text content as the data content to be analyzed.

[0224] Optionally, the identification unit 1310 can also be used to, when it is detected that the electronic device displays a target interface, obtain the image content of the content displayed in the target interface as a second image and obtain the text content of the content displayed in the target interface as a second text content through a graphic and text extraction framework in response to a confirmation operation of a target user on a target control; and use the second image and the second text content as the data content to be analyzed.

[0225] Optionally, the identification unit 1310 can also be used to, when it is detected that the electronic device displays a target interface, respond to a confirmation operation of the target user on a target control, take a screenshot of the display content in the target interface to obtain a third image; and use the third image as the data content to be analyzed.

[0226] Optionally, the recognition unit 1310 can also be used to, when it is detected that the electronic device displays a target interface, perform text recognition on the displayed content in the target interface through optical character recognition in response to a confirmation operation of the target user on the target control to obtain a third text content; perform a screenshot operation on the image content in the displayed content in the target interface to generate a fourth image; and use the third text content and the fourth image as the data content to be analyzed.

[0227] Optionally, the identification unit 1310 can also be used to detect whether the interface displayed by the electronic device matches a pre-set reference interface; when the interface displayed by the electronic device matches the reference interface, it is determined that the electronic device is detected to display a target interface; when the electronic device is detected to display the target interface, in response to the target user's confirmation operation on the target control, a text recognition operation and an image recognition operation are performed on the display content in the target interface to obtain the data content to be analyzed.

[0228] Optionally, the identification unit 1310 can also be used to identify scene information corresponding to the interface displayed by the electronic device; determine a reference interface corresponding to the scene information based on pre-set scene interface control data, wherein the scene interface control data includes at least one standard scene information and a standard interface corresponding to each standard scene information; and detect whether the interface displayed by the electronic device matches the reference interface.

[0229] Optionally, the identification unit 1310 may also be used to obtain scene interface comparison data;

[0230] In response to the target user's update operation on the scene interface control data, the specified scene information determined by the target user and the specified interface corresponding to the specified scene information are obtained; the specified scene information and the specified interface corresponding to the specified scene information are respectively added to the scene interface control data as new standard scene information and a new standard interface matched by the new standard scene information to update the scene interface control data; based on the updated scene interface control data, a reference interface matched by the scene information is determined.

[0231] The text generation unit 1320 is used to perform semantic analysis on the data content to be analyzed by using a large model algorithm to obtain target text information as the input content of the target user.

[0232] Optionally, the text generation unit 1320 can also be used to display a second interactive control; in response to the target user's confirmation operation on the second interactive control, the target text information is filled into an input box in the target interface for receiving the input content of the target user.

[0233] Optionally, the text generation unit 1320 may also be configured to store the target text information and generate prompt information when the target text information is not successfully filled into an input box in the target interface for receiving input content of the target user.

[0234] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0235] In several embodiments provided in the present application, the coupling between the units may be electrical, mechanical or other forms of coupling.

[0236] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0237] See also Fig.14 , Fig.14 A cloud server 1400 provided in an embodiment of the present application is shown, and the cloud server 1400 is connected to the electronic device 110 .

[0238] The cloud server 1400 can be used to obtain the data content to be analyzed obtained by the electronic device 110 based on the text generation method shown in the above embodiments, and perform semantic analysis on the data content to be analyzed through the large model algorithm to obtain the target text information as the input content of the target user. For a detailed introduction, please refer to the above method embodiments, which will not be repeated here.

[0239] Among them, the large model algorithm can be deployed in the cloud server.

[0240] See also Fig.15 , Fig.15 The structural block diagram of an electronic device provided in an embodiment of the present application is shown. The electronic device 110 may be a smart phone or a tablet computer, etc. The electronic device 110 in the present application may include one or more of the following components: a processor 111, a memory 112, and one or more application programs, wherein the processor 111 is electrically connected to the memory 112, and the one or more program configurations are used to execute the methods described in the aforementioned embodiments.

[0241] The processor 111 may include one or more processing cores. The processor 111 uses various interfaces and lines to connect various parts of the entire electronic device 110, and executes various functions and processes data of the electronic device 110 by running or executing instructions, programs, code sets or instruction sets stored in the memory 112, and calling data stored in the memory 112. Optionally, the processor 111 can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor 111 can integrate one or a combination of a central processing unit, an image processor, and a modem. Among them, the CPU mainly processes the operating system, the user interface, and the computer program; the GPU is responsible for rendering and drawing the display content; and the modem is used to process wireless communication. It can be understood that the above-mentioned modem may not be integrated into the processor 111, but may be implemented separately through a communication chip. Specifically, the method described in the above embodiment can be executed by one or more processors 111.

[0242] For some embodiments, the memory 112 may include a random access memory (RAM) or a read-only memory (ROM). The memory 112 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 112 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the following various method embodiments, etc. The data storage area may also store data created by the electronic device 110 during use, etc.

[0243] Please refer to Fig.16 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 1600 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0244] Computer readable storage medium 1600 can be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read Only Memory), EPROM, hard disk or ROM. Optionally, computer readable storage medium 1600 includes non-transitory computer-readable storage medium. Computer readable storage medium 1600 has storage space for program code 1610 that executes any method step in the above method. These program codes can be read from one or more computer program products or written into one or more computer program products. Program code 1610 can be compressed, for example, in an appropriate form.

[0245] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A text generation method, characterized in that: Applied to electronic equipment, the method comprises: In the case where it is detected that the electronic device displays a target interface, in response to a confirmation operation of a target user on a target control, a text recognition operation and an image recognition operation are performed on the display content in the target interface to obtain data content to be analyzed, wherein the target interface is used to receive input content of the target user, and the data content to be analyzed includes at least one of text content and image content; The data content to be analyzed is semantically analyzed by a large model algorithm to obtain target text information as input content of the target user.

2. The method according to claim 1, characterized in that The target control includes a first interactive control, and in the case of detecting that the electronic device displays a target interface, in response to a confirmation operation of a target user on the target control, performing a text recognition operation and an image recognition operation on display content in the target interface to obtain data content to be analyzed, including: In the case where it is detected that the electronic device displays a target interface, displaying a first interactive control at a designated position of the target interface; In response to a confirmation operation of the target user on the first interactive control, a text recognition operation and an image recognition operation are performed on the display content in the target interface to obtain data content to be analyzed.

3. The method according to claim 2, characterized in that The method of displaying a first interactive control at a designated position of the target interface upon detecting that the electronic device displays the target interface comprises: When detecting that the electronic device displays a target interface, obtaining a display condition of a virtual keyboard in the target interface; If it is detected that the virtual keyboard has been displayed, a first interactive control is displayed at a specified position of the target interface.

4. The method according to claim 3, characterized in that: The designated position includes a position above the display area of ​​the virtual keyboard and adjacent to the virtual keyboard.

5. The method according to claim 1, characterized in that: After the semantic analysis of the data content to be analyzed is performed by the large model algorithm to obtain the target text information as the input content of the target user, the method further includes: displaying a second interactive control; In response to the target user's confirmation operation on the second interactive control, the target text information is filled into an input box in the target interface for receiving input content of the target user.

6. The method according to claim 5, characterized in that The method further comprises: In the case where the target text information is not successfully filled into the input box in the target interface for receiving the input content of the target user, the target text information is stored and prompt information is generated.

7. The method according to claim 1, characterized in that In the case where it is detected that the electronic device displays a target interface, in response to a confirmation operation of a target user on a target control, performing a text recognition operation and an image recognition operation on the display content in the target interface to obtain data content to be analyzed, including: In the case where it is detected that the electronic device displays a target interface, in response to a confirmation operation of a target user on a target control, acquiring a first uniform resource locator corresponding to the target interface; Acquire a hypertext markup language for describing the target page based on the first uniform resource locator; Extracting a second uniform resource locator corresponding to a first identifier from the hypertext markup language, wherein the first identifier is used as identity information of a first image, and the first image includes image content in the display content; acquiring a first image based on the second uniform resource locator; The text content displayed in the target page is extracted from the hypertext markup language as the first text content, and the first image and the first text content are used as the data content to be analyzed.

8. The method according to claim 1, characterized in that: In the case where it is detected that the electronic device displays a target interface, in response to a confirmation operation of a target user on a target control, performing a text recognition operation and an image recognition operation on the display content in the target interface to obtain data content to be analyzed, including: In the case where it is detected that the electronic device displays a target interface, in response to a confirmation operation of a target user on a target control, acquiring, through an image-text extraction framework, image content of content displayed in the target interface as a second image, and acquiring text content of content displayed in the target interface as a second text content; The second image and the second text content are used as the data content to be analyzed.

9. The method according to claim 1, characterized in that: In the case where it is detected that the electronic device displays a target interface, in response to a confirmation operation of a target user on a target control, performing a text recognition operation and an image recognition operation on the display content in the target interface to obtain data content to be analyzed, including: When it is detected that the electronic device displays a target interface, in response to a confirmation operation of a target user on a target control, a screenshot operation is performed on display content in the target interface to obtain a third image; The third image is used as the data content to be analyzed.

10. The method according to claim 1, characterized in that In the case where it is detected that the electronic device displays a target interface, in response to a confirmation operation of a target user on a target control, performing a text recognition operation and an image recognition operation on the display content in the target interface to obtain data content to be analyzed, including: In the case where it is detected that the electronic device displays a target interface, in response to a confirmation operation of a target user on a target control, performing text recognition on display content in the target interface through optical character recognition to obtain third text content; Performing a screenshot operation on the image content in the display content in the target interface to generate a fourth image; The third text content and the fourth image are used as the data content to be analyzed.

11. The method according to claim 1, characterized in that In the case where it is detected that the electronic device displays a target interface, in response to a confirmation operation of a target user on a target control, performing a text recognition operation and an image recognition operation on the display content in the target interface to obtain data content to be analyzed, including: Detecting whether the interface displayed by the electronic device matches a preset reference interface; When the interface displayed by the electronic device matches the reference interface, determining that the electronic device is detecting displaying a target interface; When it is detected that the electronic device displays a target interface, in response to a confirmation operation of a target user on a target control, a text recognition operation and an image recognition operation are performed on the display content in the target interface to obtain data content to be analyzed.

12. The method according to claim 11, characterized in that The detecting whether the interface displayed by the electronic device matches a preset reference interface includes: Identifying scene information corresponding to the interface displayed by the electronic device; Determine a reference interface corresponding to the scene information based on preset scene interface reference data, wherein the scene interface reference data includes at least one standard scene information and a standard interface corresponding to each standard scene information; Detect whether the interface displayed by the electronic device matches the reference interface.

13. The method according to claim 12, characterized in that The determining of a reference interface corresponding to the scene information based on pre-set scene interface comparison data includes: Get scene interface comparison data; In response to the target user's update operation on the scene interface comparison data, obtaining the specified scene information determined by the target user and the specified interface corresponding to the specified scene information; The designated scene information and the designated interface corresponding to the designated scene information are added to the scene interface comparison data as new standard scene information and a new standard interface matched with the new standard scene information, respectively, to update the scene interface comparison data; Based on the updated scene interface comparison data, a reference interface matching the scene information is determined.

14. A text generation device, characterized in that: Applied to electronic equipment, the device comprises: an identification unit, configured to, when detecting that the electronic device displays a target interface, perform a text recognition operation and an image recognition operation on the display content in the target interface in response to a confirmation operation of a target user on a target control, to obtain data content to be analyzed, wherein the target interface is used to receive input content of a target user, and the data content to be analyzed includes at least one of text content and image content; The text generation unit is used to perform semantic analysis on the data content to be analyzed by using a large model algorithm to obtain target text information as input content of the target user.

15. A cloud server, characterized in that: The cloud server is used to obtain the data content to be analyzed obtained by the electronic device based on the method described in any one of claims 1 to 13, and perform semantic analysis on the data content to be analyzed through a large model algorithm to obtain target text information as input content of the target user.

16. An electronic device, characterized in that: include: one or more processors; Memory; One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1-13.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, and the program code can be called by a processor to execute the method according to any one of claims 1 to 13.

Images