Screen content recognition method, device, equipment and storage medium

Abstract

The present application discloses a method, apparatus, device and storage medium for screen content recognition, and belongs to the field of computer technology. The method comprises: if a screen content recognition instruction is received, then obtaining n trajectory lines drawn on the screen of the electronic device, where n is a positive integer. Afterwards, obtaining a designated mark according to the n trajectory lines, the designated mark comprising a first mark and / or a second mark, the first mark being used to indicate content that needs to be recognized, and the second mark being used to indicate content that does not need to be recognized. Identifying target content from the screen content of the electronic device according to the designated mark. The present application does not need to recognize the entire screen content of the electronic device, but can recognize target content that meets the user's needs from the screen content of the electronic device according to the designated mark drawn by the user. The recognition time is short, content recognition can be completed quickly, and power consumption can be saved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, device, and storage medium for identifying screen content. Background Art

[0002] With the development of computer technology, electronic devices such as mobile phones, tablets, and laptops have become an indispensable part of people's daily lives. Users can use these devices to browse content. When browsing content such as images on these devices, users may need to recognize the text contained in them.

[0003] In the related art, when recognizing text in an image, all the text in the image is recognized. However, recognizing the entire text not only takes a long time but also increases power consumption. Summary of the Invention

[0004] This application provides a method, apparatus, device, and storage medium for screen content recognition, which can shorten recognition time and save power consumption. The technical solution is as follows:

[0005] In a first aspect, a screen content recognition method is provided. The method is applied to an electronic device. In this method, upon receiving a screen content recognition instruction, n trajectory lines drawn on the screen of the electronic device are obtained, where n is a positive integer. Subsequently, designated markers are obtained based on the n trajectory lines, and target content is recognized from the screen content of the electronic device based on the designated markers.

[0006] Optionally, the screen content may be an image or interface displayed by an electronic device, and the interface may be an application interface, a video playback interface, or a camera preview interface, but is not limited thereto. In other words, the present application can be applied to identifying content in various images or interfaces displayed by electronic devices, and can be applied in a variety of scenarios to facilitate user use.

[0007] Optionally, the designated mark includes a first mark and / or a second mark, the first mark is used to indicate content that needs to be identified, and the second mark is used to indicate content that does not need to be identified.

[0008] As an example, the first marker is a closed figure, and the second marker is an open figure. In this case, the first marker is used to circle the content to be recognized, and the second marker is used to block the content that does not need to be recognized. Optionally, the user can freely draw a trajectory line on the screen to ultimately draw the first and second markers.

[0009] As another example, the first mark is a first preset graphic, the second mark is a second preset graphic, and the first preset graphic and the second preset graphic have different shapes.

[0010] For example, the first preset figure is a closed figure, and the second preset figure is also a closed figure. For example, the first preset figure can be a convex polygon, and the second preset figure can be a concave polygon, or the first preset figure and the second preset figure can be convex polygons of different shapes. In this case, the first mark is used to circle the content that needs to be recognized, and the second mark is used to circle the content that does not need to be recognized.

[0011] For another example, the first preset graphic is a closed graphic and the second preset graphic is an open graphic. In this case, the first mark is used to circle the content that needs to be recognized, and the second mark is used to block the content that does not need to be recognized.

[0012] Optionally, the electronic device may provide a first graphic option and a second graphic option. After the user selects the first graphic option, the user may draw a trajectory line of the first preset graphic on the screen. After the user selects the second graphic option, the user may draw a trajectory line of the second preset graphic on the screen.

[0013] As another example, the first mark is a closed figure, the second mark is a closed figure, and the line thickness of the first mark and the second mark is different. In this case, the first mark is used to circle the content that needs to be recognized, and the second mark is used to circle the content that does not need to be recognized.

[0014] Optionally, the electronic device may provide a first line option and a second line option. After the user selects the first line option, they can draw a trajectory line of a first thickness on the screen. After the user selects the second preset pattern, they can draw a trajectory line of a second thickness on the screen. The first thickness and the second thickness are different, the first thickness being the line thickness of the first mark, and the second thickness being the line thickness of the second mark.

[0015] As another example, the first mark is a closed figure, the second mark is a closed figure, and the line colors of the first mark and the second mark are different. In this case, the first mark is used to circle the content that needs to be recognized, and the second mark is used to circle the content that does not need to be recognized.

[0016] Optionally, the electronic device may provide a first color option and a second color option. After the user selects the first color option, the user can draw a trajectory line of the first color on the screen. After the user selects the second color option, the user can draw a trajectory line of the second color on the screen. The first color and the second color are different, the first color being the line color of the first mark, and the second color being the line color of the second mark.

[0017] In this application, the user can draw a trajectory line on the screen, and the electronic device can determine the designated mark (i.e., the first mark and / or the second mark) based on the trajectory line drawn by the user, and then can determine which contents in the screen need to be identified and which contents do not need to be identified based on the designated marks, and then identify the target content that meets the user's needs from the screen content.

[0018] In this application, there is no need to identify the entire screen content of the electronic device. Instead, the target content that meets the user's needs can be identified from the screen content of the electronic device based on the designated mark drawn by the user. The recognition time is short, the content recognition can be completed quickly, and power consumption can be saved. In addition, since the user can choose the content to be recognized or not recognized according to their own needs, that is, the user can select the content in a local area of ​​the screen content to be recognized or not, or can select the content in multiple discontinuous areas of the screen content to be recognized or not, the present application can realize cross-line recognition of screen content, thereby making the recognition of screen content more flexible.

[0019] Optionally, the first mark is a closed figure, and the second mark is a non-closed figure. The operation of obtaining the specified mark based on the n trajectory lines can be: performing polygon fitting processing on the n trajectory lines; if a polygon is fitted through at least one trajectory line among the n trajectory lines, the fitted polygon is determined as the first mark, and the trajectory lines among the n trajectory lines that do not fit the polygon are determined as the second mark; if multiple polygons are fitted through at least one trajectory line among the n trajectory lines, the first mark is determined based on the overlap between the multiple fitted polygons and the multiple polygons, and the trajectory lines among the n trajectory lines that do not fit the polygon are determined as the second mark; if no polygon is fitted through the n trajectory lines, the n trajectory lines are determined as the second mark.

[0020] Since the trajectory lines freely drawn by users are usually not so accurate and regular, this application performs polygon fitting on the drawn trajectory lines to obtain relatively regular polygons, which can restore the user's drawing intentions to a certain extent, thereby facilitating the accurate determination of the first mark and the second mark, and thus accurately determining the user's recognition needs.

[0021] Optionally, the first mark is used to circle content that needs to be identified, and the second mark is used to circle content that does not need to be identified or to block content that does not need to be identified. The operation of identifying the target content from the screen content of the electronic device based on the specified mark can be: if the specified mark includes the first mark and does not include the second mark, then the target content is identified from the screen content of the electronic device based on the first area circled by the first mark; if the specified mark includes the second mark and does not include the first mark, then the target content is identified from the screen content of the electronic device based on the second area circled by the second mark or based on the content blocked by the second mark.

[0022] In this application, the target content can be identified from the screen content of the electronic device based on the first area circled by the first marker, or the target content can be identified from the screen content of the electronic device based on the second area circled by the second marker or based on the content blocked by the second marker. In this way, the entire article does not need to be recognized in this application, but specific target content can be selected for recognition based on user needs. The recognition time is short, content recognition can be completed quickly, and power consumption can be saved.

[0023] Optionally, the operation of identifying target content from the screen content of the electronic device according to the first area circled by the first mark may include the following four cases:

[0024] The first case: If the content located in the first area of ​​the screen content of the electronic device is text content, then the text paragraph in which the text content in the first area is located is determined to be the first paragraph. Based on the coordinates of each sentence in the first paragraph and the coordinate range of the first area, the overlap between each sentence in the first paragraph and the first area is determined, and based on the overlap between each sentence in the first paragraph and the first area, a target sentence is identified from the first paragraph, and the target sentence is the target content; or, based on the coordinates of each character in the first paragraph and the coordinate range of the first area, the overlap between each character in the first paragraph and the first area is determined, and based on the overlap between each character in the first paragraph and the first area, a target character is identified from the first paragraph, and the target character is the target content.

[0025] In this application, the actual outlined area (i.e., the first area) can be combined with the text paragraph, and then the sentence coordinates or character coordinates of the text paragraph are used to analyze the overlap between the sentences or characters in the text paragraph and the outlined area, thereby determining the target sentence or target character to be recognized. In this way, even if the user outlines an incomplete sentence or only half a character, the user's outline intention can be restored to a certain extent, thereby accurately identifying the target sentence or target character that meets the user's needs.

[0026] The second case: if the content located in the first area of ​​the screen content of the electronic device is table content, then the table to which the table content in the first area belongs is determined to be the first table; based on the coordinates of each cell in the first table and the coordinate range of the first area, the overlap degree between each cell in the first table and the first area is determined; based on the overlap degree between each cell in the first table and the first area, the text in the target cell is identified from the first table, and the text in the target cell is the target content.

[0027] In this application, the actual outlined area (i.e., the first area) can be combined with a table, and then the cell coordinates of the table are used to analyze the overlap between the cells in the table and the outlined area, thereby determining the target cell to be identified. In this way, even if the user does not outline the entire cell, the user's outline intention can be restored to a certain extent, thereby accurately identifying the text in the target cell that meets the user's needs.

[0028] The third case: if the content in the first area of ​​the screen content of the electronic device is picture content, the picture to which the picture content in the first area belongs is determined to be the first picture; the first picture is identified according to the coordinates of the first picture, and the first picture is the target content.

[0029] In this application, users can select pictures by drawing a first mark on the picture to achieve the effect of identifying the picture, thereby improving the flexibility of screen content recognition.

[0030] The fourth situation: if the content in the first area of ​​the screen content of the electronic device is formula content, the formula in which the formula content in the first area is located is identified, and the formula in which the formula content in the first area is located is the target content.

[0031] In this application, the user can select a check mark. The user can achieve the effect of identifying the formula by drawing the first mark on the formula, thereby improving the flexibility of screen content recognition.

[0032] Optionally, the first mark is used to circle the content that needs to be identified, and the second mark is used to circle the content that does not need to be identified or to block the content that does not need to be identified. The operation of identifying the target content from the screen content of the electronic device according to the specified mark can be: if the specified mark includes the first mark and the second mark, and the first mark and the second mark do not overlap, then the first mark is retained in the specified mark and the second mark is discarded, and the target content is identified from the screen content of the electronic device according to the first area circled by the first mark. Alternatively, if the specified mark includes the first mark and the second mark, and the first mark and the second mark do not overlap, then the first mark is discarded in the specified mark and the second mark is retained, and the target content is identified from the screen content of the electronic device according to the second area circled by the second mark or according to the content blocked by the second mark.

[0033] In this application, if both the first mark and the second mark exist but do not overlap, then the delineation intentions of the two marks conflict. In this case, one of the two marks can be discarded and the other can be retained so that content recognition can continue normally.

[0034] Optionally, the first mark is used to circle content that needs to be identified, and the second mark is used to circle content that does not need to be identified. The operation of identifying target content from the screen content of the electronic device based on the designated mark can be: if the designated mark includes the first mark and the second mark, and the second mark is located within the first mark, then the other areas of the first area circled by the first mark except the second area circled by the second mark are determined as third areas, and the target content is identified from the screen content of the electronic device based on the third area.

[0035] In this application, if the second mark is located within the first mark, it means that the user wants to identify the content of other areas (i.e., the third area) in the first area circled by the first mark except for the second area circled by the second mark. Therefore, the target content can be identified from the screen content of the electronic device based on the third area, and the target content identified in this way meets the user needs.

[0036] Optionally, the first mark is used to circle content to be identified, and the second mark is used to block content that does not need to be identified. The operation of identifying target content from the screen content of the electronic device based on the specified mark can be: if the specified mark includes the first mark and the second mark, and the second mark is located within the first mark, then the target content is identified from the screen content of the electronic device based on the first area circled by the first mark, and the content blocked by the second mark is deleted from the identified target content.

[0037] In this application, if the second mark is located within the first mark, it means that the user wants to identify the content in the first area circled by the first mark except for the content blocked by the second mark. Therefore, the target content can be identified from the screen content of the electronic device based on the first area circled by the first mark, and then the content blocked by the second mark can be deleted from the identified target content. The target content obtained in this way meets the user's needs.

[0038] Optionally, the first mark is used to circle the content that needs to be identified, and the second mark is used to circle the content that does not need to be identified. The operation of identifying the target content from the screen content of the electronic device according to the specified mark can be: if the specified mark includes a first mark and a second mark, and the first mark intersects the second mark, then retain the first mark and use the intersection of the first mark and the second mark as the new second mark, or retain the second mark and use the union of the second mark and the first mark as the new first mark. The other areas in the first area circled by the first mark except the second area circled by the second mark are determined as the third area, and the target content is identified from the screen content of the electronic device according to the third area.

[0039] In this application, if the first mark intersects the second mark, it means that the user is likely to want to identify the content of the area (i.e., the third area) in the first area circled by the first mark except the intersection area of ​​the first mark and the second mark. Therefore, the target content can be identified from the screen content of the electronic device based on the third area, and the target content identified in this way meets the user needs.

[0040] Optionally, the first mark is used to circle the content that needs to be identified, and the second mark is used to block the content that does not need to be identified. The operation of identifying the target content from the screen content of the electronic device according to the specified mark can be: if the specified mark includes the first mark and the second mark, and the first mark intersects the second mark, then retain the first mark and use the part of the second mark located within the first mark as the new second mark; identify the target content from the screen content of the electronic device according to the first area circled by the first mark, and delete the content blocked by the second mark from the identified target content.

[0041] In this application, if the first mark intersects the second mark, it means that the user is likely to want to identify the content in the first area circled by the first mark except for the content partially obscured by the second mark within the first mark. Therefore, the target content can be identified from the screen content of the electronic device based on the first area circled by the first mark, and then the content partially obscured by the second mark within the first mark can be deleted from the identified target content. The target content obtained in this way meets the user's needs.

[0042] Furthermore, after identifying target content from the screen content of the electronic device based on the designated marker, the identified target content can be extracted. Specifically, the identified target content can be stored in the system clipboard. Subsequently, if any edit box is detected to have focus, the target content stored in the system clipboard is pasted into the edit box.

[0043] The present application can realize cross-line recognition of the screen content of the electronic device, so after extracting the identified target content, the cross-line extraction of the screen content of the electronic device is realized, thereby improving the flexibility of screen content extraction. In addition, since the edit box that obtains the focus is most likely the edit box that the user currently needs to use, the embodiment of the present application can directly paste the identified target content into the edit box that obtains the focus, thereby facilitating user use and improving the user experience.

[0044] In a second aspect, a screen content recognition device is provided, wherein the screen content recognition device has the function of implementing the screen content recognition method described in the first aspect. The screen content recognition device includes at least one module, wherein the at least one module is configured to implement the screen content recognition method described in the first aspect.

[0045] In a third aspect, a screen content recognition device is provided. The device comprises a processor and a memory, wherein the memory is configured to store a program that supports the device in executing the screen content recognition method provided in the first aspect, and to store data used to implement the screen content recognition method described in the first aspect. The processor is configured to execute the program stored in the memory. The device may further comprise a communication bus configured to establish a connection between the processor and the memory.

[0046] In a fourth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium. When the computer-readable storage medium is run on a computer, the computer executes the screen content recognition method described in the first aspect.

[0047] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the screen content recognition method described in the first aspect.

[0048] The technical effects obtained by the above-mentioned second, third, fourth and fifth aspects are similar to the technical effects obtained by the corresponding technical means in the above-mentioned first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0050] Figure 2 This is a block diagram of a software system of an electronic device provided in an embodiment of the present application;

[0051] Figure 3 This is a schematic diagram of an image display provided by an embodiment of the present application;

[0052] Figure 4 This is a schematic diagram of a trajectory line provided in an embodiment of the present application;

[0053] Figure 5 This is a schematic diagram of an application interface provided in an embodiment of the present application;

[0054] Figure 6 is a schematic diagram of a video playback interface provided in an embodiment of the present application;

[0055] Figure 7 is a schematic diagram of a camera preview interface provided in an embodiment of the present application;

[0056] Figure 8 Schematic diagram of the first mark and the second mark provided in the embodiment of the present application;

[0057] Figure 9 is a schematic diagram of a first preset graph provided in an embodiment of the present application;

[0058] Figure 10 is a schematic diagram of a second preset graph provided in an embodiment of the present application;

[0059] Figure 11 is a schematic diagram of a second first mark and a second mark provided in an embodiment of the present application;

[0060] Figure 12 This is a flow chart of a screen content recognition method provided by an embodiment of the present application;

[0061] Figure 13 This is a schematic diagram of a notification bar provided in an embodiment of the present application;

[0062] Figure 14 is a schematic diagram of a graphic option provided in an embodiment of the present application;

[0063] Figure 15 This is a schematic diagram of polygon fitting and merging provided in an embodiment of the present application;

[0064] Figure 16 is a schematic diagram of another trajectory line provided in an embodiment of the present application;

[0065] Figure 17 is a schematic diagram of a first region provided in an embodiment of the present application;

[0066] Figure 18 is a schematic diagram of an editing box provided in an embodiment of the present application;

[0067] Figure 19 is a schematic diagram of a table provided in an embodiment of the present application;

[0068] Figure 20 is a schematic diagram of an image correction provided by an embodiment of the present application;

[0069] Figure 21 This is a schematic diagram of formula recognition provided by an embodiment of the present application;

[0070] Figure 22 is a schematic diagram of another table provided in an embodiment of the present application;

[0071] Figure 23 This is a flowchart of the processing of the first first mark and the second mark provided in an embodiment of the present application;

[0072] Figure 24 This is a schematic diagram of the processing of the first first mark and the second mark provided in an embodiment of the present application;

[0073] Figure 25 is a processing flow chart of the second first mark and second mark provided in an embodiment of the present application;

[0074] Figure 26 This is a schematic diagram of processing the second first mark and the second mark provided in an embodiment of the present application;

[0075] Figure 27 This is a flowchart of the processing of the third first mark and second mark provided in an embodiment of the present application;

[0076] Figure 28 This is a schematic diagram of processing the third first mark and second mark provided in an embodiment of the present application;

[0077] Figure 29 This is a flowchart of the fourth processing of the first mark and the second mark provided in an embodiment of the present application;

[0078] Figure 30 This is a schematic diagram of processing the fourth first mark and second mark provided in an embodiment of the present application;

[0079] Figure 31 This is a flowchart of the fifth processing of the first mark and the second mark provided in an embodiment of the present application;

[0080] Figure 32 This is a schematic diagram of the fifth processing of the first mark and the second mark provided in an embodiment of the present application;

[0081] Figure 33 This is a flow chart of video content recognition provided by an embodiment of the present application;

[0082] Figure 34 Schematic diagram of the third first mark and second mark provided in an embodiment of the present application;

[0083] Figure 35 is a schematic diagram of a screen content recognition method provided by an embodiment of the present application;

[0084] Figure 36 is a flowchart of another screen content recognition method provided by an embodiment of the present application;

[0085] Figure 37 This is a schematic diagram of the first content identification processing process provided by an embodiment of the present application;

[0086] Figure 38 is a schematic diagram of the second content identification processing process provided by an embodiment of the present application;

[0087] Figure 39 is a schematic diagram of the third content identification processing process provided in an embodiment of the present application;

[0088] Figure 40 is a schematic diagram of the fourth content identification processing process provided by an embodiment of the present application;

[0089] Figure 41 It is a structural diagram of a screen content recognition device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0090] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0091] It should be understood that the “multiple” mentioned in this application refers to two or more. In the description of this application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; “and / or” in this article is merely a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate the clear description of the technical solution of this application, words such as “first” and “second” are used to distinguish between identical or similar items with basically the same functions and effects. Those skilled in the art can understand that words such as “first” and “second” do not limit the quantity and execution order, and words such as “first” and “second” do not necessarily limit them to be different.

[0092] The phrases "one embodiment" or "some embodiments" described in this application mean that the specific features, structures, or characteristics described in that embodiment are included in one or more embodiments of the application. Thus, the phrases "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" that appear in different places in this application do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. In addition, the terms "including," "comprising," "having," and their variations all mean "including but not limited to," unless otherwise specifically emphasized.

[0093] The electronic device involved in the embodiments of the present application is described below.

[0094] Figure 1 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Figure 1The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a SIM card interface 195, etc. Among them, the sensor module 180 can include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0095] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0096] The processor 110 may include one or more processing units, for example, an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0097] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0098] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0099] The charging management module 140 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also provide power to the electronic device 100 via the power management module 141.

[0100] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0101] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0102] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0103] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0104] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0105] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0106] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0107] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created by the electronic device 100 during use (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0108] The electronic device 100 can implement audio functions, such as music playback and recording, through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D and the application processor.

[0109] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0110] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device 100 by inserting it into or removing it from the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is an integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0111] Next, the software system of the electronic device 100 will be described.

[0112] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a microservice architecture, or a cloud architecture. In the embodiment of the present application, the software system of the electronic device 100 is exemplarily described by taking the Android system of the layered architecture as an example.

[0113] Figure 2 This is a block diagram of a software system of an electronic device 100 provided in an embodiment of the present application. Figure 2 The layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other via software interfaces. In some embodiments, the Android system includes an application layer, an application framework layer, an Android runtime layer, a system layer, and a kernel layer.

[0114] The application layer can include a series of applications. Figure 2 As shown, the application may include a camera, a gallery, a calendar, a call, a map, a navigation, a WLAN, a Bluetooth, a short message, and the like.

[0115] The application framework layer provides application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Figure 2As shown, the application framework layer may include a window manager, content provider, view system, telephony manager, resource manager, notification manager, and so on. The window manager is used to manage window programs. It can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, and so on. The content provider is used to store and retrieve data and make this data accessible to applications. This data may include video, images, audio, incoming and outgoing calls, browsing history and bookmarks, and the phone book. The view system includes visual controls, such as those for displaying text and images. The view system can be used to construct the application's display interface, which may consist of one or more views, such as a view that displays a text message notification icon, a view that displays text, and a view that displays images. The telephony manager is used to provide communication functions for electronic device 100, such as managing call status (including connected and ended calls). The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, and video files. The notification manager enables applications to display notification information in the status bar. This information can be used to convey notification messages and can be displayed briefly and then disappear automatically without user interaction. For example, a notification manager is used to notify users of completed downloads, message alerts, and more. A notification manager can also appear as an icon or scrolling text bar in the system's top status bar, such as notifications from background applications. A notification manager can also appear as a dialog window on the screen, such as a text message in the status bar, a beep, a vibration on an electronic device, or a flashing indicator light.

[0116] The Android Runtime consists of core libraries and a virtual machine (VM). The Android runtime is responsible for scheduling and management of the Android system. The core library consists of two parts: one for Java-based functions and the other for the Android core library. The application layer and application framework layer run in the VM. The VM executes Java files from the application layer and application framework layer as binary files. The VM is responsible for performing functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0117] The system library can include multiple functional modules, such as: surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc. The surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing. The 2D graphics engine is a drawing engine for 2D drawing.

[0118] The kernel layer is the layer between hardware and software. The kernel layer includes display drivers, camera drivers, audio drivers, sensor drivers, etc.

[0119] The application scenarios involved in the embodiments of the present application are described below.

[0120] With the development of computer technology, electronic devices such as mobile phones, tablet computers, and laptops have become an indispensable part of people's daily lives. Users can use electronic devices to browse content. In some scenarios, such as Figure 3 As shown, when a user browses images and other content in an electronic device, he or she may need to identify and extract the text contained therein.

[0121] In related technologies, when identifying and extracting text from an image, all text within the image is recognized and extracted. Specifically, all text within the image is first recognized, and after recognition is complete, a text extraction icon is displayed. Users can click the text extraction icon to perform extraction operations on the recognized text, such as copying, translating, sharing, and searching.

[0122] However, in the above method, on the one hand, the recognition of the entire text will result in a longer recognition time. Considering the number of words and the computing power, the time to complete the recognition may even be as long as several seconds, which will cause the display of the text extraction icon to lag. On the other hand, the recognition of the entire text will also lead to increased power consumption. In addition, since the entire text is ultimately recognized, and the user can only select text continuously, it is impossible for the user to select text across lines contained in the image.

[0123] To this end, the embodiment of the present application provides a screen content recognition method that allows users to flexibly select the content to be recognized and / or not recognized in the screen content. Figure 4 As shown in Figure (a), the user browses images on the mobile phone. In this case, Figure 4As shown in Figure (b) in the figure, the user can draw a trajectory line on the screen of the mobile phone. This trajectory line can circle area A in the image to indicate that the user chooses to recognize the content in area A instead of recognizing the content in other areas. Or, Figure 4 As shown in Figure (c) in the figure, the user can draw a trajectory line on the screen of the mobile phone. This trajectory line can circle area B in the image to indicate that the user chooses not to recognize the content in area B, but to recognize the content in other areas. In this way, in the embodiment of the present application, there is no need to recognize the entire article, but the content in a specific area can be selected according to user needs. The recognition time is short, the content recognition can be completed quickly, and power consumption can be saved. In addition, since the user can choose the area to be recognized or not according to his own needs, cross-line recognition of screen content can be achieved, so that the subsequent user selection of the recognized content is to realize cross-line selection of screen content, thereby making the recognition and extraction of screen content more flexible.

[0124] The following describes the screen content involved in the embodiments of the present application.

[0125] The screen content recognition method provided in the embodiments of the present application can be used to recognize the screen content of an electronic device. The electronic device can be a mobile phone, tablet computer, laptop computer, television, conference equipment, etc., which is not limited in the embodiments of the present application. The screen content is the content displayed by the electronic device.

[0126] Alternatively, as Figure 3 As shown, the screen content may be an image displayed by the electronic device. In this case, the screen content recognition method provided in the embodiment of the present application may be used to recognize the content contained in the image displayed by the electronic device.

[0127] Optionally, the screen content may be an interface displayed by an electronic device, such as an application interface, a video playback interface, a camera preview interface, etc., but not limited thereto. In this case, the screen content recognition method provided in the embodiment of the present application can be used to identify the content contained in the interface displayed by the electronic device.

[0128] For example, Figure 5 As shown, the screen content may be an application interface 501 of an information application displayed on an electronic device, and the application interface 501 of the information application is used to display information. In this case, the information information displayed in the application interface 501 of the information application can be identified by the screen content recognition method provided in the embodiment of the present application.

[0129] For example, Figure 6As shown, the screen content may be a video playback interface 601 of a video application displayed by an electronic device, and the video playback interface 601 of the video application is used to play videos. Before executing the screen content recognition method provided in the embodiment of the present application, the video played in the video playback interface 601 of the video application may be in a paused state. In this case, the screen content recognition method provided in the embodiment of the present application can be used to recognize the video image of the video in the paused state in the video playback interface 601 of the video application.

[0130] For example, Figure 7 As shown, the screen content may be a camera preview interface 701 of a camera application displayed on an electronic device, where the camera preview interface 701 of the camera application is used to display a preview image captured by a camera. In this case, the preview image displayed in the camera preview interface 701 of the camera application can be identified using the screen content recognition method provided in an embodiment of the present application.

[0131] Optionally, the screen content recognition method provided in the embodiment of the present application is applicable to recognizing text in various fonts contained in the screen content. For example, it can recognize text in printed, typed, handwritten, written and other fonts contained in the screen content. The embodiment of the present application is not limited to this.

[0132] The following describes the designated marks involved in the embodiments of the present application.

[0133] The screen content recognition method provided in the embodiments of the present application can determine the content to be recognized and / or the content not to be recognized based on the designated mark drawn by the user on the screen of the electronic device when recognizing the screen content of the electronic device. The designated mark involved in the embodiments of the present application includes a first mark and / or a second mark. The first mark is used to indicate the content to be recognized, and the second mark is used to indicate the content not to be recognized.

[0134] The first marker and the second marker are different so that the electronic device can distinguish the first marker from the second marker, thereby identifying target content that meets the user's needs from the screen content based on the first marker and the second marker. Several possible implementations of the first marker and the second marker are described below:

[0135] In a first possible implementation, the first mark may be a closed figure, and the second mark may be an open figure. In this case, the first mark is used to enclose the content to be identified, and the second mark is used to block the content not to be identified.

[0136] Optionally, the user may freely draw a trajectory line on the screen to ultimately draw the first mark and the second mark.

[0137] For example, Figure 8As shown in Figure (a), the first mark can be a closed figure formed by the trajectory line A freely drawn by the user, and the content circled by the first mark is the content to be identified. Figure 8 As shown in FIG. 5( b ), the second mark may be a non-closed figure formed by a trajectory line B freely drawn by the user, and the content blocked by the second mark is content that does not need to be recognized.

[0138] In a second possible implementation, the first mark may be a predefined first preset graphic, and the second mark may be a predefined second preset graphic. The first preset graphic and the second preset graphic have different shapes.

[0139] As an example, the first preset figure is a closed figure, and the second preset figure is also a closed figure. For example, the first preset figure can be a convex polygon, and the second preset figure can be a concave polygon; or the first preset figure and the second preset figure can be convex polygons of different shapes. In this case, the first mark is used to circle the content that needs to be recognized, and the second mark is used to circle the content that does not need to be recognized.

[0140] As another example, the first preset graphic is a closed graphic, and the second preset graphic is an open graphic. In this case, the first mark is used to circle the content that needs to be recognized, and the second mark is used to block the content that does not need to be recognized.

[0141] Optionally, the electronic device may provide multiple graphic options. The user can select different graphic options to draw different graphic trajectories on the screen, ultimately drawing the first mark and the second mark. For example, the electronic device may provide a first graphic option and a second graphic option. After the user selects the first graphic option, the user can draw a trajectory of a first preset graphic on the screen. After the user selects the second graphic option, the user can draw a trajectory of a second preset graphic on the screen.

[0142] In some embodiments, the first preset graphic may include multiple predefined graphics. In this case, the electronic device may provide multiple first graphic options, each of which corresponds to the multiple graphics. After the user selects one of the multiple first graphic options, the user can draw a trajectory line of the graphic corresponding to the first graphic option on the screen.

[0143] In some embodiments, the second preset graphic may include multiple predefined graphics. In this case, the electronic device may provide multiple second graphic options, each corresponding to the multiple graphics. After the user selects one of the multiple second graphic options, the user can draw a trajectory line of the graphic corresponding to the second graphic option on the screen.

[0144] For example, the first preset graphic may include a predefined Figure 9 At least one of the three closed figures shown in Figures (a), (b), and (c) in FIG, but not limited to these. The second preset figure may include a predefined Figure 10 At least one of the three non-closed figures shown in Figures (a), (b), and (c), but not limited to these.

[0145] In a third possible implementation, the first mark can be a closed figure, and the second mark can also be a closed figure, but the line thickness of the first mark and the second mark are different. The line thickness of the first mark can be pre-set, and the line thickness of the second mark can also be pre-set. In this case, the first mark is used to circle the content that needs to be recognized, and the second mark is used to circle the content that does not need to be recognized.

[0146] Optionally, the electronic device may provide two line options. The user can freely draw trajectory lines of different thicknesses on the screen by selecting different line options to ultimately draw a first mark and a second mark. For example, the electronic device may provide a first line option and a second line option. After the user selects the first line option, the user can draw a trajectory line of the first thickness on the screen. After the user selects the second preset pattern, the user can draw a trajectory line of the second thickness on the screen. The first thickness and the second thickness are different, the first thickness is the line thickness of the first mark, and the second thickness is the line thickness of the second mark.

[0147] For example, Figure 11 As shown in Figure (a), the first mark can be a closed figure formed by the trajectory line A freely drawn by the user, and the content circled by the first mark is the content to be identified. Figure 11 As shown in Figure (b), the second marker can be a closed figure formed by a freely drawn trajectory line B by the user. The content enclosed by the second marker is the content that does not need to be recognized. The line thickness of trajectory line B is different from that of trajectory line A.

[0148] In a fourth possible implementation, the first mark can be a closed figure, and the second mark can also be a closed figure, but the line colors of the first mark and the second mark are different. The line color of the first mark can be pre-set, and the line color of the second mark can also be pre-set. In this case, the first mark is used to circle the content that needs to be recognized, and the second mark is used to circle the content that does not need to be recognized.

[0149] Optionally, the electronic device may provide two color options. The user can freely draw different colored trajectory lines on the screen by selecting different color options to ultimately draw a first mark and a second mark. For example, the electronic device may provide a first color option and a second color option. After the user selects the first color option, the user can draw a trajectory line of the first color on the screen. After the user selects the second color option, the user can draw a trajectory line of the second color on the screen. The first color and the second color are different, the first color is the line color of the first mark, and the second color is the line color of the second mark.

[0150] For example, the line color of the first mark can be green, and the line color of the second mark can be red. In this case, the first mark can be a closed figure formed by a green trajectory line freely drawn by the user, and the content circled by the first mark is the content to be recognized. The second mark can be a closed figure formed by a red trajectory line freely drawn by the user, and the content circled by the second mark is the content not to be recognized.

[0151] It should be noted that in an embodiment of the present application, the user can draw a trajectory line on the screen, and the electronic device can determine the designated mark (i.e., the first mark and / or the second mark) based on the trajectory line drawn by the user, and then can determine which contents in the screen need to be identified and which contents do not need to be identified based on the designated marks, and then identify the target content that meets the user's needs from the screen content.

[0152] The screen content recognition method provided in the embodiment of the present application is explained in detail below.

[0153] Figure 12 This is a flow chart of a screen content recognition method provided by an embodiment of the present application. The method can be applied to an electronic device, which can be the above Figures 1 to 2 The electronic device described in the embodiment. Figure 12 , the method comprises the following steps:

[0154] Step 1201: If the electronic device receives a screen content recognition instruction, the electronic device obtains n trajectory lines drawn on the screen of the electronic device, where n is a positive integer.

[0155] The screen content recognition instruction is used to instruct the screen content of the electronic device to be recognized. The screen content recognition instruction can be triggered by the user in the electronic device. The screen content recognition instruction can be triggered in a variety of ways. In one possible way, if the electronic device detects the selection operation of the screen recognition button, it is determined that the screen content recognition instruction has been received. For example, Figure 13As shown, the user can slide down on the screen of the electronic device to open the notification bar 1301 of the electronic device. The notification bar 1301 may display a WLAN button, a mobile data button, a mute button, an auto-rotate button, a Bluetooth button, a screen recognition button 1302, etc. After the user clicks the selected screen recognition button, the screen content recognition instruction will be triggered in the electronic device, and the electronic device will receive the screen content recognition instruction. After that, the user can draw a track line on the screen of the electronic device, and the electronic device can obtain the track line drawn by the user on the screen of the electronic device.

[0156] In a first possible implementation, if the difference between the first mark and the second mark is that the first mark is a closed figure and the second mark is a non-closed figure, for example, the first mark and the second mark are Figure 8 If the method shown is implemented, then after the user triggers the screen content recognition command in the electronic device, the user can freely draw a trajectory line on the screen of the electronic device to draw a closed figure and / or a non-closed figure. During the process of the user drawing the trajectory line on the screen, the electronic device can obtain the trajectory line drawn by the user on the screen of the electronic device.

[0157] In a second possible implementation, if the difference between the first mark and the second mark is that the first mark is a first preset graphic and the second mark is a second preset graphic, for example, the first mark is Figure 9 The second mark is implemented in the manner shown. Figure 10 If the method shown is implemented, then after the user triggers the screen content recognition instruction in the electronic device, the electronic device can display a first graphic option and a second graphic option. In this case, after the user selects the first graphic option, the user can draw a trajectory line of the first preset graphic on the screen of the electronic device. After the user selects the second graphic option, the user can draw a trajectory line of the second preset graphic on the screen of the electronic device. While the user draws the trajectory line on the screen, the electronic device can obtain the trajectory line drawn by the user on the screen of the electronic device.

[0158] For example, after the user clicks the selected screen recognition button in the electronic device, Figure 14 As shown in FIG. 1 (a), the electronic device may display the first graphic option 1401 and the second graphic option 1402 in a floating manner on top of the current screen content. Figure 14 As shown in FIG. 14( b ), the user can click on the first graphic option 1401, and then the user can draw a trajectory line A of the first preset graphic on the screen of the electronic device. Figure 14 As shown in FIG. 14( c ), the user may click on the second graphic option 1402 , and then the user may draw a trajectory line B of the second preset graphic on the screen of the electronic device.

[0159] In a third possible implementation, if the difference between the first mark and the second mark is that the first mark is a closed figure and the second mark is also a closed figure, but the line thickness of the first mark and the second mark is different, for example, the first mark and the second mark are Figure 11 If it is implemented in the manner shown, then after the user triggers the screen content recognition instruction in the electronic device, the electronic device can display a first line option and a second line option. In this case, after the user selects the first line option, the user can draw a trajectory line of the first thickness on the screen of the electronic device. After the user selects the second line option, the user can draw a trajectory line of the second thickness on the screen of the electronic device, wherein the first thickness is different from the second thickness, the first thickness is the line thickness of the first mark, and the second thickness is the line thickness of the second mark. During the process of the user drawing the trajectory line on the screen, the electronic device can obtain the trajectory line drawn by the user on the screen of the electronic device.

[0160] In a fourth possible implementation, if the difference between the first mark and the second mark is that the first mark is a closed figure and the second mark is also a closed figure, but the line colors of the first mark and the second mark are different, then after the user triggers the screen content recognition instruction in the electronic device, the electronic device can display the first color option and the second color option. For example, after the user clicks the selected screen recognition button in the electronic device, the electronic device can display the first color option and the second color option in a floating manner above the current screen content. In this case, after the user selects the first color option, the user can draw a trajectory line of the first color on the screen of the electronic device. After the user selects the second color option, the user can draw a trajectory line of the second color on the screen of the electronic device, wherein the first color is different from the second color, the first color is the line color of the first mark, and the second color is the line color of the second mark. In the process of the user drawing the trajectory line on the screen, the electronic device can obtain the trajectory line drawn by the user on the screen of the electronic device.

[0161] Optionally, when a user draws a trajectory line on the screen of an electronic device, after drawing some trajectory lines, the user can adjust the size of the area enclosed by the drawn trajectory line or the size of the area blocked by the drawn trajectory line by dragging or other methods. For example, after a user draws a trajectory line of a closed figure on the screen of an electronic device, the user can adjust the size of the drawn closed figure by dragging or other methods to adjust the size of the area enclosed by the closed figure. For another example, after a user draws a trajectory line of a non-closed figure on the screen of an electronic device, the user can adjust the size of the drawn non-closed figure by dragging or other methods to adjust the size of the area blocked by the non-closed figure.

[0162] As an example, the electronic device may further provide an end recognition button. After the user has completed the outline, he or she may click the end recognition button to trigger the end recognition instruction. At this time, the selection of the screen recognition button will be canceled. After the electronic device receives the end recognition instruction, it may obtain a specified mark (including a first mark and / or a second mark) based on all the track lines (i.e., n track lines) drawn by the user on the screen of the electronic device, that is, it may execute the following step 1202. For example, the end recognition button may be displayed in a drop-down notification bar of the electronic device, or the end recognition button may be displayed in the form of a floating button on the upper layer of the screen content of the electronic device. Of course, the end recognition button may also be displayed in other forms, which is not limited in the embodiments of the present application.

[0163] As another example, after completing the drawing, the user can click the screen recognition button again to deselect the screen recognition button, which will also trigger the end recognition instruction. After receiving the end recognition instruction, the electronic device can obtain designated marks based on all the trajectory lines drawn by the user on the screen of the electronic device, that is, it can execute the following step 1202.

[0164] As another example, after the user clicks the selected screen recognition button for a preset duration, the electronic device can automatically deselect the screen recognition button, thereby triggering an end recognition instruction. After receiving the end recognition instruction, the electronic device can obtain designated markers based on all traces drawn by the user on the screen of the electronic device, thereby executing step 1202.

[0165] It should be noted that in the embodiments of the present application, a user can draw a track line on the screen of an electronic device in a variety of ways. For example, a user can draw a track line on the screen of an electronic device by touching the screen with a finger, or a user can draw a track line on the screen of an electronic device with a stylus, or a user can draw a track line on the screen of an electronic device by using a mouse or keyboard, or a user can draw a track line on the screen of an electronic device by voice operation, somatosensory operation, air gesture operation, etc. Of course, a user can also draw a track line on the screen of an electronic device in other ways, and the embodiments of the present application are not limited to this.

[0166] Optionally, the trajectory line drawn by the user on the screen of the electronic device can be displayed on the screen of the electronic device. That is, after the electronic device obtains the trajectory line drawn by the user on the screen of the electronic device, it can display the trajectory line on the screen. In this case, after receiving the end recognition instruction, the electronic device can also perform a screenshot operation to obtain a screenshot, which includes the screen content displayed by the electronic device and the n trajectory lines drawn by the user for the screen content. In this case, the electronic device can save the screenshot, and further, the electronic device can also remind the user to share the screenshot. The user can share the screenshot with others through the electronic device.

[0167] Step 1202: The electronic device obtains a designated mark according to the n trajectory lines.

[0168] The user draws a trajectory line on the screen of the electronic device to indicate the content that needs to be recognized and / or the content that does not need to be recognized. Therefore, after the electronic device obtains the n trajectory lines drawn by the user on the screen of the electronic device, it can obtain a designated mark based on the n trajectory lines, so that it can subsequently determine which content in the screen content needs to be recognized and which content does not need to be recognized.

[0169] In a first possible implementation, if the difference between the first mark and the second mark is that the first mark is a closed figure and the second mark is a non-closed figure, then the operation of step 1202 may include the following steps (1) to (4):

[0170] (1) The electronic device performs polygon fitting processing on the n trajectory lines.

[0171] Polygon fitting refers to fitting an irregular curve with a polygon instead of a straight line, thereby fitting the curve into a regular polygon. In the embodiment of the present application, polygon fitting is performed on the n trajectory lines, which attempts to fit one or more polygons through one or more trajectory lines in the n trajectory lines.

[0172] It is worth noting that since the trajectory lines freely drawn by users are usually not so accurate and regular, the embodiment of the present application performs polygon fitting on the drawn trajectory lines to obtain relatively regular polygons, which can restore the user's drawing intention to a certain extent, thereby facilitating the accurate determination of the first mark and the second mark, and thus accurately determining the user's recognition needs.

[0173] (2) If a polygon is fitted by at least one of the n trajectory lines, the electronic device determines the fitted polygon as a first mark, and determines the trajectory line of the n trajectory lines that does not fit a polygon as a second mark.

[0174] If a polygon is fitted through at least one of the n trajectory lines, it means that the at least one trajectory line forms a closed figure (ie, the polygon), and thus the formed polygon can be determined as the first mark.

[0175] In this case, the other trajectory lines among the n trajectory lines except the at least one trajectory line are trajectory lines that are not fitted with a polygon. These trajectory lines that are not fitted with a polygon form a non-closed figure, so these trajectory lines that are not fitted with a polygon can be determined as the second mark.

[0176] In particular, if all the n trajectory lines are fitted with a polygon, that is, if there is no trajectory line among the n trajectory lines that is not fitted with a polygon, then there is no need to determine the second mark. In this case, the designated marks obtained by the electronic device based on the n trajectory lines only include the first mark.

[0177] (3) If multiple polygons are fitted by at least one of the n trajectory lines, the electronic device determines a first mark based on the multiple fitted polygons and the overlap of the multiple polygons, and determines the trajectory line of the n trajectory lines that does not fit a polygon as a second mark.

[0178] The electronic device may determine the first marker based on the fitted multiple polygons and the overlap between the multiple polygons by: if the multiple polygons do not overlap with each other, then the multiple polygons are determined as the first marker; if there are overlapping polygons among the multiple polygons, then the overlapping polygons are merged to obtain a new polygon, and the merged new polygon and the polygons among the multiple polygons that do not overlap with other polygons are determined as the first marker.

[0179] If multiple polygons are fitted by at least one of the n trajectory lines, it means that the at least one trajectory line forms multiple closed figures (i.e., the multiple polygons), and thus the first marker can be determined based on the overlap between the multiple polygons. For any two polygons in the multiple polygons, since both polygons are used to indicate the content to be identified, if the two polygons overlap, the two polygons can be merged to obtain a new polygon as the first marker, and the enclosed area of ​​the new polygon is the union of the enclosed areas of the two polygons.

[0180] In this case, the other trajectory lines among the n trajectory lines except the at least one trajectory line are trajectory lines that are not fitted with a polygon. These trajectory lines that are not fitted with a polygon form a non-closed figure, so these trajectory lines that are not fitted with a polygon can be determined as the second mark.

[0181] In particular, if all the n trajectory lines are fitted with a polygon, that is, if there is no trajectory line among the n trajectory lines that is not fitted with a polygon, then there is no need to determine the second mark. In this case, the designated marks obtained by the electronic device based on the n trajectory lines only include the first mark.

[0182] (4) If no polygon is fitted through the n trajectory lines, the electronic device determines the n trajectory lines as second marks.

[0183] If no polygon is fitted through the n trajectory lines, it means that the n trajectory lines form an open figure, and thus the n trajectory lines can be determined as the second mark. In this case, the designated mark obtained by the electronic device based on the n trajectory lines only includes the second mark.

[0184] For example, Figure 15 As shown, the electronic device first performs polygon fitting on the n trajectory lines, and fits multiple polygons through at least one trajectory line among the n trajectory lines, for example, Figure 16 As shown in Figure (a), the electronic device obtains the trajectory lines 1, 2, 3, and 4 drawn on the screen of the electronic device. After the electronic device performs polygon fitting on these four trajectory lines, it fits the trajectory line 1 to obtain the trajectory line 4. Figure 16 The polygon 1 shown in Figure (b) is fitted by the trajectory line 2. Figure 16 The polygon 2 shown in Figure (b) is fitted by the trajectory line 3. Figure 16 The polygon 3 shown in FIG. (b) is obtained, while the trajectory line 4 does not fit a polygon. Afterwards, the electronic device merges the multiple polygons according to the overlap of the multiple polygons fitted to obtain a new polygon, for example, Figure 16 As shown in Figure (b), polygon 1 and polygon 2 overlap, then polygon 1 and polygon 2 can be merged to obtain Figure 16 The electronic device then determines the merged new polygon and the polygons that do not overlap with other polygons as the first mark, and determines the trajectory line that does not fit the polygon in the n trajectory lines as the second mark. For example, Figure 16 The new polygon 4 and polygon 3 shown in Figure (c) are determined as the first mark, and the trajectory line 4 is determined as the second mark.

[0185] In a second possible implementation, if the difference between the first mark and the second mark is that the first mark is a first preset graphic and the second mark is a second preset graphic, then the operation of step 1202 may be: for any one of the n trajectory lines, if the graphic formed by this trajectory line is the first preset graphic, then the electronic device determines the graphic formed by this trajectory line as the first mark; if the graphic formed by this trajectory line is the second preset graphic, then the electronic device determines the graphic formed by this trajectory line as the second mark.

[0186] In this implementation, each of the n trajectory lines drawn by the user on the screen of the electronic device is directly a trajectory line of the first preset pattern or a trajectory line of the second preset pattern. That is, the pattern formed by each of the n trajectory lines is directly the first preset pattern or the second preset pattern. Therefore, the electronic device can directly determine the first mark and the second mark from the pattern formed by the n trajectory lines. In this case, the process of the electronic device obtaining the first mark and the second mark is relatively simple, which can improve the acquisition efficiency.

[0187] In a third possible implementation, if the difference between the first mark and the second mark is that the first mark is a closed figure and the second mark is also a closed figure, but the line thickness of the first mark and the second mark is different, then the operation of step 1202 may include the following steps (1) to (3):

[0188] (1) The electronic device groups the n trajectory lines to obtain a first group of trajectory lines and / or a second group of trajectory lines.

[0189] The line thickness of the track lines in the first group of track lines is the line thickness of the first marker, and the line thickness of the track lines in the second group of track lines is the line thickness of the second marker.

[0190] In particular, if the line thickness of each of the n trajectory lines is the line thickness of the first marker, then after the electronic device groups the n trajectory lines, only the first group of trajectory lines is obtained. In this case, there is no second marker, and the designated markers obtained by the electronic device based on the n trajectory lines only include the first marker.

[0191] If the line thickness of each of the n trajectory lines is the line thickness of the second marker, then after the electronic device groups the n trajectory lines, only the second group of trajectory lines is obtained. In this case, the first marker does not exist, and the designated markers obtained by the electronic device based on the n trajectory lines only include the second marker.

[0192] (2) The electronic device performs polygon fitting processing on the first set of trajectory lines. If a polygon is fitted by at least one trajectory line in the first set of trajectory lines, the electronic device determines the fitted polygon as a first marker. If multiple polygons are fitted by at least one trajectory line in the first set of trajectory lines, the electronic device determines the first marker based on the multiple polygons and the overlap of the multiple polygons. If no polygon is fitted by all the trajectory lines in the first set of trajectory lines, the electronic device determines that the first marker does not exist.

[0193] In the embodiment of the present application, performing polygon fitting processing on the first group of trajectory lines is attempting to fit one or more polygons through one or more trajectory lines in the first group of trajectory lines.

[0194] If a polygon is fitted by at least one trajectory line in the first group of trajectory lines, it means that the at least one trajectory line forms a closed figure (ie, the polygon), and thus the formed polygon can be determined as the first marker.

[0195] The electronic device may determine the first marker based on the fitted multiple polygons and the overlap between the multiple polygons by: if the multiple polygons do not overlap with each other, then the multiple polygons are determined as the first marker; if there are overlapping polygons among the multiple polygons, then the overlapping polygons are merged to obtain a new polygon, and the merged new polygon and the polygons among the multiple polygons that do not overlap with other polygons are determined as the first marker.

[0196] If multiple polygons are fitted by at least one of the first set of trajectory lines, it means that the at least one trajectory line forms multiple closed figures (i.e., the multiple polygons), and thus the first marker can be determined based on the overlap between the multiple polygons. For any two polygons in the multiple polygons, since both polygons are used to indicate the content to be identified, if the two polygons overlap, the two polygons can be merged to obtain a new polygon as the first marker, and the enclosed area of ​​the new polygon is the union of the enclosed areas of the two polygons.

[0197] If no polygon is fitted by the first set of trajectory lines, it means that the first set of trajectory lines form non-closed figures, and thus it can be determined that the first mark does not exist.

[0198] (3) The electronic device performs polygon fitting processing on the second set of trajectory lines. If a polygon is fitted by at least one trajectory line in the second set of trajectory lines, the electronic device determines the fitted polygon as a second marker. If multiple polygons are fitted by at least one trajectory line in the second set of trajectory lines, the electronic device determines the second marker based on the multiple polygons and the overlap of the multiple polygons. If no polygon is fitted by all the trajectory lines in the second set of trajectory lines, the electronic device determines that the second marker does not exist.

[0199] In the embodiment of the present application, the polygon fitting process is performed on the second group of trajectory lines, which is an attempt to fit one or more polygons through one or more trajectory lines in the second group of trajectory lines.

[0200] If a polygon is fitted by at least one trajectory line in the second group of trajectory lines, it means that the at least one trajectory line forms a closed figure (ie, the polygon), and thus the formed polygon can be determined as the second marker.

[0201] The electronic device may determine the second marker based on the fitted multiple polygons and the overlap between the multiple polygons by: if the multiple polygons do not overlap with each other, then the multiple polygons are determined as the second marker; if there are overlapping polygons among the multiple polygons, then the overlapping polygons are merged to obtain a new polygon, and the merged new polygon and the polygons among the multiple polygons that do not overlap with other polygons are determined as the second marker.

[0202] If multiple polygons are fitted by at least one of the second set of trajectory lines, it indicates that the at least one trajectory line forms multiple closed figures (i.e., the multiple polygons), and thus the second marker can be determined based on the overlap between the multiple polygons. For any two polygons in the multiple polygons, since both polygons are used to indicate content that does not need to be recognized, if the two polygons overlap, the two polygons can be merged to obtain a new polygon as the second marker, and the enclosed area of ​​the new polygon is the union of the enclosed areas of the two polygons.

[0203] If no polygon is fitted by the second set of trajectory lines, it means that the second set of trajectory lines form non-closed figures, and thus it can be determined that the second mark does not exist.

[0204] In a fourth possible implementation, if the difference between the first mark and the second mark is that the first mark is a closed figure and the second mark is also a closed figure, but the line colors of the first mark and the second mark are different, then the operation of step 1202 may include the following steps (1) to (3):

[0205] (1) The electronic device groups the n trajectory lines to obtain a first group of trajectory lines and / or a second group of trajectory lines.

[0206] The line color of the track lines in the first group of track lines is the line color of the first marker, and the line color of the track lines in the second group of track lines is the line color of the second marker.

[0207] In particular, if the line color of each of the n trajectory lines is the line color of the first marker, then after the electronic device groups the n trajectory lines, only the first group of trajectory lines is obtained. In this case, there is no second marker, and the designated markers obtained by the electronic device based on the n trajectory lines only include the first marker.

[0208] If the line color of each of the n trajectory lines is the line color of the second marker, then after the electronic device groups the n trajectory lines, only the second group of trajectory lines is obtained. In this case, the first marker does not exist, and the designated markers obtained by the electronic device based on the n trajectory lines only include the second marker.

[0209] (2) The electronic device performs polygon fitting processing on the first set of trajectory lines. If a polygon is fitted by at least one trajectory line in the first set of trajectory lines, the electronic device determines the fitted polygon as a first marker. If multiple polygons are fitted by at least one trajectory line in the first set of trajectory lines, the electronic device determines the first marker based on the multiple polygons and the overlap of the multiple polygons. If no polygon is fitted by all the trajectory lines in the first set of trajectory lines, the electronic device determines that the first marker does not exist.

[0210] (3) The electronic device performs polygon fitting processing on the second set of trajectory lines. If a polygon is fitted by at least one trajectory line in the second set of trajectory lines, the electronic device determines the fitted polygon as a second marker. If multiple polygons are fitted by at least one trajectory line in the second set of trajectory lines, the electronic device determines the second marker based on the multiple polygons and the overlap of the multiple polygons. If no polygon is fitted by all the trajectory lines in the second set of trajectory lines, the electronic device determines that the second marker does not exist.

[0211] Step 1203: The electronic device identifies target content from the screen content of the electronic device according to the designated mark.

[0212] In the embodiment of the present application, it is not necessary to recognize the entire screen content of the electronic device. Instead, the target content that meets the user's needs can be identified from the screen content of the electronic device based on the designated mark drawn by the user. The recognition time is short, the content recognition can be completed quickly, and power consumption can be saved. In addition, since the user can choose the content to be recognized or not recognized according to their own needs, that is, the user can select the content in a local area of ​​the screen content to be recognized or not, or can select the content in multiple discontinuous areas of the screen content to be recognized or not, the embodiment of the present application can realize cross-line recognition of screen content, thereby making the recognition of screen content more flexible.

[0213] It should be noted that the designated mark obtained by the electronic device may include one or more first marks and one or more second marks. In other words, the user can draw one or more first marks on the screen to indicate one or more content that needs to be recognized. The user can also draw one or more second marks on the screen to indicate one or more content that does not need to be recognized.

[0214] Optionally, the first mark is used to enclose content that needs to be identified, in which case the first mark is a closed figure. The second mark is used to enclose content that does not need to be identified, in which case the second mark is a closed figure. Alternatively, the second mark is used to block content that does not need to be identified, in which case the second mark is an open figure.

[0215] In this case, the operation of step 1203 may include the following three methods:

[0216] In a first manner, if the designated mark includes the first mark but does not include the second mark, the electronic device identifies the target content from the screen content of the electronic device according to the area circled by the first mark (which may be referred to as the first area).

[0217] The operation of the electronic device identifying the target content from the screen content of the electronic device according to the first area circled by the first mark may include the following four situations:

[0218] The first case: If the content located in the first area of ​​the screen content of the electronic device is text content, the electronic device determines that the text paragraph in which the text content in the first area is located is the first paragraph. The electronic device determines the degree of overlap between each sentence in the first paragraph and the first area based on the coordinates of each sentence in the first paragraph and the coordinate range of the first area, and identifies the target sentence from the first paragraph based on the degree of overlap between each sentence in the first paragraph and the first area, and the target sentence is the target content; or the electronic device determines the degree of overlap between each character in the first paragraph and the first area based on the coordinates of each character in the first paragraph and the coordinate range of the first area, and identifies the target character from the first paragraph based on the degree of overlap between each character in the first paragraph and the first area, and the target character is the target content.

[0219] The electronic device can first determine the coordinate range of the first area circled by the first mark, and then detect the content in the coordinate range of the first area in the screen content of the electronic device to determine whether the content in the first area is text content, table content, image content, or formula content. If the content in the first area is text content, the electronic device performs text line detection on the screen content of the electronic device to determine the coordinates of each text line in the screen content of the electronic device (i.e., the coordinates of the starting position and the coordinates of the ending position of the text line), and then performs text paragraph analysis based on the coordinates of each text line in the screen content of the electronic device to obtain the coordinates of each text paragraph in the screen content of the electronic device (i.e., the coordinates of the starting position and the coordinates of the ending position of the text paragraph) and the coordinates of each text line in each text paragraph. Afterwards, the electronic device can determine the text paragraph (i.e., the first paragraph) in which the text content in the first area is located based on the coordinate range of the first area, the coordinates of each text paragraph in the screen content of the electronic device, and the coordinates of each text line in each text paragraph. At this time, the coordinates of the first paragraph and the coordinates of each text line in the first paragraph are also determined.

[0220] As an example, the electronic device can determine the coordinates of each sentence in the first paragraph. For example, the electronic device can use a segmentation algorithm in natural language processing (NLP) to determine the coordinates of each sentence in the first paragraph. The electronic device can then determine the overlap between each sentence in the first paragraph and the first area based on the coordinates of each sentence in the first paragraph and the coordinate range of the first area. Optionally, the overlap between a sentence in the first paragraph and the first area can be the ratio of the overlapping area of ​​the sentence (i.e., the area where the sentence overlaps with the first area) to the total area of ​​the sentence. That is, the overlap between the sentence and the first area represents the proportion of the area of ​​the sentence that falls within the first area. For any sentence in the first paragraph, if the overlap between the sentence and the first area is greater than or equal to the first overlap, the electronic device can determine the sentence as a target sentence and recognize the sentence. The first overlap can be set in advance, and the first overlap can be set larger. For example, the first overlap can be 50%, 60%, etc., which is not limited in the embodiments of the present application.

[0221] As another example, the electronic device can determine the coordinates of each character in the first paragraph. For example, the character may include letters, numbers, Chinese characters, symbols, etc. The electronic device can then determine the degree of overlap between each character in the first paragraph and the first area based on the coordinates of each character in the first paragraph and the coordinate range of the first area. Optionally, the degree of overlap between a certain character in the first paragraph and the first area can be the ratio of the overlapping area of ​​the character (i.e., the area where the character overlaps with the first area) to the total area of ​​the character, that is, the degree of overlap between the character and the first area represents the proportion of the area of ​​the character that falls within the first area. For any character in the first paragraph, if the degree of overlap between the character and the first area is greater than or equal to the first degree of overlap, the electronic device can determine the character as the target character and recognize the character.

[0222] For example, Figure 17As shown, the electronic device can determine the coordinates of each character in the text paragraph (i.e., the first paragraph) where the text content in the first area is located. Then, the electronic device can determine the overlap degree of each character in the first paragraph with the first area according to the coordinates of each character in the first paragraph and the coordinate range of the first area. In this case, since the overlap degree of other characters in the first paragraph except the three characters "大", "之", and "组" with the first area is 100%, the electronic device can determine other characters in the first paragraph except the three characters "大", "之", and "组" as target characters and identify these characters. Since the overlap degree of the character "组" in the first paragraph with the first area is greater than 50%, the electronic device can determine the character "组" in the first paragraph as a target character and identify this character. However, since the overlap degrees of the two characters "大" and "之" in the first paragraph with the first area are both less than 50%, the electronic device does not identify the two characters "大" and "之" in the first paragraph.

[0223] It should be noted that in the embodiments of the present application, the actual drawn area (i.e., the first area) can be combined with the text paragraph, and then the overlap situation between the sentences or characters in the text paragraph and the drawn area can be analyzed by using the sentence coordinates or character coordinates of the text paragraph, so as to determine the target sentences or target characters to be recognized accordingly. In this way, even when the user's drawing is not a complete sentence or only half a character, the user's drawing intention can be restored to a certain extent, so that the target sentences or target characters that meet the user's needs can be accurately recognized.

[0224] Further, after the electronic device recognizes the target sentence from the first paragraph, it also obtains the encoding information of the target sentence. In this case, the electronic device can also extract the recognized target sentence. Optionally, the electronic device can store the recognized target sentence (i.e., the encoding information of the target sentence) in the system clipboard.

[0225] Similarly, after the electronic device recognizes the target character from the first paragraph, it also obtains the encoding information of the target character. In this case, the electronic device can also extract the recognized target character. Optionally, the electronic device can store the recognized target character (i.e., the encoding information of the target character) in the system clipboard.

[0226] It should be noted that since the embodiments of the present application can achieve cross-line recognition of the screen content of the electronic device, cross-line extraction of the screen content of the electronic device is achieved after extracting the recognized target content, thereby improving the flexibility of screen content extraction.

[0227] The system clipboard is a data storage area used to temporarily store data that may need to be pasted later.

[0228] As an example, if an edit box exists in the screen content of the electronic device, the electronic device can directly paste the target content (target sentence or target character) stored in the system clipboard into the edit box.

[0229] As another example, if an edit box exists in the screen content of the electronic device and the edit box obtains focus, the electronic device may paste the target content stored in the system clipboard into the edit box.

[0230] It is worth noting that the edit box that obtains focus is most likely the edit box that the user currently needs to use. Therefore, in the embodiment of the present application, the identified target content can be directly pasted into the edit box that obtains focus, thereby facilitating user use and improving user experience.

[0231] As another example, after the electronic device detects a paste instruction in any edit box, it can paste the target content stored in the system clipboard into the edit box.

[0232] As another example, after detecting that any edit box has obtained focus, the electronic device may paste the target content stored in the system clipboard into the edit box.

[0233] For example, Figure 18 As shown in FIG. 18( a ), after the user draws a first mark on the screen content 1801 of the electronic device, the electronic device stores the target content in the first area circled by the first mark to the system clipboard. Figure 18 As shown in FIG. 18( b ), the user switches from the screen content 1801 of the electronic device to another interface 1802, in which there is an edit box. After the user clicks on the edit box in the interface 1802, the edit box gets the focus. In this case, Figure 18 As shown in Figure (c), the electronic device will automatically paste the target content stored in the system clipboard into the edit box.

[0234] In the second scenario, if the content in the first area of ​​the screen content of the electronic device is table content, the electronic device determines that the table to which the table content in the first area belongs is the first table. The electronic device determines the degree of overlap between each cell in the first table and the first area based on the coordinates of each cell in the first table and the coordinate range of the first area; and identifies text in a target cell in the first table based on the degree of overlap between each cell in the first table and the first area, where the text in the target cell is the target content.

[0235] The electronic device can first determine the coordinate range of the first area circled by the first mark, and then detect the content in the coordinate range of the first area in the screen content of the electronic device to determine whether the content in the first area is text content, table content, image content, or formula content. If the content in the first area is table content, the electronic device analyzes the table structure of the table (i.e., the first table) to which the table content in the first area belongs to obtain the coordinates of each cell in the first table. The electronic device can then determine the overlap between each cell in the first table and the first area based on the coordinates of each cell in the first table and the coordinate range of the first area. Optionally, the overlap between a cell in the first table and the first area can be the ratio of the overlapping area of ​​the cell (i.e., the area where the cell overlaps with the first area) to the total area of ​​the cell, that is, the overlap between the cell and the first area represents the proportion of the area of ​​the cell that falls within the first area. For any cell in the first table, if the overlap between the cell and the first area is greater than or equal to the second overlap, the electronic device can determine the cell as a target cell and recognize the text in the cell. The second overlap degree may be pre-set, for example, the second overlap degree may be 30%, 40%, etc., which is not limited in the embodiment of the present application.

[0236] In some embodiments, after the electronic device determines the target cells based on the degree of overlap between each cell in the first table and the first area, it can also align the coordinates of all the determined target cells in the first table, that is, align all the determined target cells up and down and left and right in the first table to obtain a sub-table with a row and b columns containing all the determined target cells from the first table, where a and b are both positive integers.

[0237] Afterwards, the electronic device may recognize text in a target cell in the subtable. In this case, the target cell in the identified subtable contains text, while the other cells are empty. Alternatively, the electronic device may determine all cells in the subtable as target cells and then recognize text in all cells in the subtable. In this case, all cells in the identified subtable contain text.

[0238] For example, Figure 19As shown in Figure (a), the electronic device can determine the coordinates of each cell in the table (i.e., the first table) to which the table content in the first area belongs. The electronic device can then determine the degree of overlap between each cell in the first table and the first area based on the coordinates of each cell in the first table and the coordinate range of the first area. In this case, since the degree of overlap between the cell in the 2nd row and 2nd column, the cell in the 2nd row and 3rd column, and the cell in the 3rd row and 2nd column in the first table and the first area is greater than 30%, the electronic device determines these three cells as target cells. Afterwards, the electronic device aligns the coordinates of these three target cells in the first table to obtain a sub-table with 3 rows and 2 columns from the first table. As an example, the electronic device recognizes the text in the target cell in the sub-table, and the recognized sub-table can be as follows Figure 19 As shown in FIG. (b) in FIG. 1 , in this case, the target cell in the identified subtable contains text, while the other cells are empty. As another example, the electronic device determines all cells in the subtable as target cells and recognizes the text in all cells in the subtable. The identified subtable can be as follows: Figure 19 As shown in Figure (c), in this case, all cells in the identified subtable contain text.

[0239] In the embodiment of the present application, the actual outlined area (i.e., the first area) can be combined with a table, and then the cell coordinates of the table are used to analyze the overlap between the cells in the table and the outlined area, thereby determining the target cell to be identified. In this way, even if the user does not outline the entire cell, the user's outline intention can be restored to a certain extent, thereby accurately identifying the text in the target cell that meets the user's needs.

[0240] Furthermore, after the electronic device identifies the subtable, it also obtains the encoding information of the subtable. In this case, the electronic device can also extract the identified subtable. Optionally, the electronic device can store the identified subtable (i.e., the encoding information of the subtable) to the system clipboard.

[0241] As an example, if an edit box exists in the screen content of the electronic device, the electronic device may directly paste the sub-table stored in the system clipboard into the edit box.

[0242] As another example, if an edit box exists in the screen content of the electronic device and the edit box obtains focus, the electronic device may paste the sub-table stored in the system clipboard into the edit box.

[0243] As another example, after the electronic device detects a paste instruction in any edit box, it can paste the sub-table stored in the system clipboard into the edit box.

[0244] As another example, after detecting that any edit box has the focus, the electronic device may paste the subtable stored in the system clipboard into the edit box.

[0245] The third case: if the content in the first area of ​​the screen content of the electronic device is picture content, the electronic device determines that the picture to which the picture content in the first area belongs is the first picture, identifies the first picture according to the coordinates of the first picture, and the first picture is the target content.

[0246] The electronic device may first determine the coordinate range of the first area circled by the first mark, and then detect the content in the coordinate range of the first area in the screen content of the electronic device to determine whether the content in the first area is text content, table content, image content, or formula content. If the content in the first area is image content, the electronic device may analyze the image features of the image content in the first area (i.e., the first image) to which the image content belongs to obtain the coordinates of the first image. The electronic device may then identify the first image based on the coordinates of the first image.

[0247] When the electronic device identifies the first image according to the coordinates of the first image, it can also correct the identified first image to adjust the first image to normal when the first image is skewed or deformed.

[0248] For example, Figure 20 As shown in FIG. 1( a ), the electronic device can determine the coordinates of the picture (i.e., the first picture) to which the picture content in the first area belongs, and then identify the first picture according to the coordinates of the first picture, and correct the identified first picture to correct the skewed and deformed first picture to Figure 20 The normal picture shown in Figure (b).

[0249] Furthermore, after the electronic device recognizes the first image, it also obtains the encoding information of the first image. In this case, the electronic device can also extract the recognized first image. Optionally, the electronic device can store the recognized first image (i.e., the encoding information of the first image) to the system clipboard.

[0250] As an example, if an edit box exists in the screen content of the electronic device, the electronic device can directly paste the first image stored in the system clipboard into the edit box.

[0251] As another example, if an edit box exists in the screen content of the electronic device and the edit box obtains focus, the electronic device may paste the first image stored in the system clipboard into the edit box.

[0252] As another example, after the electronic device detects a paste instruction in any edit box, it can paste the first picture stored in the system clipboard into the edit box.

[0253] As another example, after detecting that any edit box has obtained focus, the electronic device may paste the first image stored in the system clipboard into the edit box.

[0254] The fourth case: if the content in the first area of ​​the screen content of the electronic device is formula content, the electronic device identifies the formula (which may be called the first formula) in which the formula content in the first area is located, and the first formula is the target content.

[0255] The electronic device may first determine the coordinate range of the first area circled by the first mark, and then detect the content within the coordinate range of the first area in the screen content of the electronic device to determine whether the content in the first area is text content, table content, image content, or formula content. If the content in the first area is formula content, the electronic device may directly identify the formula (i.e., the first formula) in which the formula content in the first area is located.

[0256] For example, Figure 21 As shown in FIG. 1(a), the electronic device can directly identify the formula (ie, the first formula) to which the formula content in the first area belongs. The identified first formula can be Figure 21 The formula shown in Figure (b) in .

[0257] Furthermore, after the electronic device recognizes the first formula, it also obtains the encoding information of the first formula. In this case, the electronic device can also extract the recognized first formula. Optionally, the electronic device can store the recognized first formula (i.e., the encoding information of the first formula) to the system clipboard.

[0258] As an example, if there is an edit box in the screen content of the electronic device, the electronic device can directly paste the first formula stored in the system clipboard into the edit box.

[0259] As another example, if an edit box exists in the screen content of the electronic device and the edit box obtains focus, the electronic device may paste the first formula stored in the system clipboard into the edit box.

[0260] As another example, after the electronic device detects a paste instruction in any edit box, it can paste the first formula stored in the system clipboard into the edit box.

[0261] As another example, after detecting that any edit box obtains focus, the electronic device may paste the first formula stored in the system clipboard into the edit box.

[0262] In the second method, if the designated mark includes the second mark but does not include the first mark, the electronic device identifies the target content from the screen content of the electronic device based on the area circled by the second mark (which may be called the second area) or based on the content blocked by the second mark.

[0263] If the second marker is used to enclose content that does not need to be recognized, the electronic device recognizes the target content from the electronic device's screen content based on the area enclosed by the second marker (i.e., the second area). If the second marker is used to block content that does not need to be recognized, the electronic device recognizes the target content from the electronic device's screen content based on the content blocked by the second marker.

[0264] The operation of the electronic device identifying the target content from the screen content of the electronic device according to the second area circled by the second mark may include the following four situations:

[0265] The first case: If the content located in the second area of ​​the screen content of the electronic device is text content, the electronic device determines that the text paragraph containing the other text content in the screen content of the electronic device except the text content in the second area is the second paragraph. The electronic device determines the degree of overlap between each sentence in the second paragraph and the second area based on the coordinates of each sentence in the second paragraph and the coordinate range of the second area, and identifies the target sentence from the second paragraph based on the degree of overlap between each sentence in the second paragraph and the second area, and the target sentence is the target content; or the electronic device determines the degree of overlap between each character in the second paragraph and the second area based on the coordinates of each character in the second paragraph and the coordinate range of the second area, and identifies the target character from the second paragraph based on the degree of overlap between each character in the second paragraph and the second area, and the target character is the target content.

[0266] The electronic device can first determine the coordinate range of the second area circled by the second mark, and then detect the content in the coordinate range of the second area in the screen content of the electronic device to determine whether the content in the second area is text content, table content, picture content, or formula content. If the content in the second area is text content, the electronic device performs text line detection on the screen content of the electronic device to determine the coordinates of each text line in the screen content of the electronic device, and then performs text paragraph analysis based on the coordinates of each text line in the screen content of the electronic device to obtain the coordinates of each text paragraph in the screen content of the electronic device and the coordinates of each text line in each text paragraph. Afterwards, the electronic device can determine the text paragraph (i.e., the second paragraph) in the screen content of the electronic device where other text content other than the text content in the second area is located based on the coordinate range of the second area, the coordinates of each text paragraph in the screen content of the electronic device, and the coordinates of each text line in each text paragraph. At this time, the coordinates of the second paragraph and the coordinates of each text line in the second paragraph are also determined.

[0267] As an example, the electronic device can determine the coordinates of each sentence in the second paragraph. For example, the electronic device can use the segmentation algorithm in NLP to determine the coordinates of each sentence in the second paragraph. The electronic device can then determine the overlap between each sentence in the second paragraph and the second area based on the coordinates of each sentence in the second paragraph and the coordinate range of the second area. Optionally, the overlap between a sentence in the second paragraph and the second area can be the ratio of the overlapping area of ​​the sentence (i.e., the area where the sentence overlaps with the second area) to the total area of ​​the sentence, that is, the overlap between the sentence and the second area represents the proportion of the area of ​​the sentence that falls within the second area. For any sentence in the second paragraph, if the overlap between the sentence and the second area is less than the first overlap, the electronic device can determine the sentence as a target sentence and identify the sentence. Among them, the first overlap can be set in advance, and the first overlap can be set larger, for example, the first overlap can be 50%, 60%, etc., and the embodiments of the present application are not limited to this.

[0268] As another example, the electronic device can determine the coordinates of each character in the second paragraph. For example, the character may include letters, numbers, Chinese characters, symbols, etc. The electronic device can then determine the degree of overlap between each character in the second paragraph and the second area based on the coordinates of each character in the second paragraph and the coordinate range of the second area. Optionally, the degree of overlap between a certain character in the second paragraph and the second area can be the ratio of the overlapping area of ​​the character (i.e., the area where the character overlaps with the second area) to the total area of ​​the character, that is, the degree of overlap between the character and the second area represents the proportion of the area of ​​the character that falls within the second area. For any character in the second paragraph, if the degree of overlap between the character and the second area is less than the first degree of overlap, the electronic device can determine the character as the target character and recognize the character.

[0269] Furthermore, after the electronic device recognizes the target sentence from the second paragraph, it also obtains the encoding information of the target sentence. In this case, the electronic device can also extract the recognized target sentence. Optionally, the electronic device can store the recognized target sentence (i.e., the encoding information of the target sentence) to the system clipboard.

[0270] Similarly, after the electronic device recognizes the target character from the second paragraph, it also obtains the encoding information of the target character. In this case, the electronic device can also extract the recognized target character. Optionally, the electronic device can store the recognized target character (i.e., the encoding information of the target character) to the system clipboard.

[0271] As an example, if an edit box exists in the screen content of the electronic device, the electronic device can directly paste the target content (target sentence or target character) stored in the system clipboard into the edit box.

[0272] As another example, if an edit box exists in the screen content of the electronic device and the edit box obtains focus, the electronic device may paste the target content stored in the system clipboard into the edit box.

[0273] As another example, after the electronic device detects a paste instruction in any edit box, it can paste the target content stored in the system clipboard into the edit box.

[0274] As another example, after detecting that any edit box has obtained focus, the electronic device may paste the target content stored in the system clipboard into the edit box.

[0275] In the second scenario, if the content in the second area of ​​the screen of the electronic device is table content, the electronic device determines that the table to which the table content in the second area belongs is the second table. The electronic device determines the degree of overlap between each cell in the second table and the second area based on the coordinates of each cell in the second table and the coordinate range of the second area; and identifies text in a target cell in the second table based on the degree of overlap between each cell in the second table and the second area, where the text in the target cell is the target content.

[0276] The electronic device can first determine the coordinate range of the second area circled by the second mark, and then detect the content in the coordinate range of the second area in the screen content of the electronic device to determine whether the content in the second area is text content, table content, image content, or formula content. If the content in the second area is table content, the electronic device analyzes the table structure of the table to which the table content in the second area belongs (i.e., the second table) to obtain the coordinates of each cell in the second table. The electronic device can then determine the overlap between each cell in the second table and the second area based on the coordinates of each cell in the second table and the coordinate range of the second area. Optionally, the overlap between a cell in the second table and the second area can be the ratio of the overlapping area of ​​the cell (i.e., the area where the cell overlaps with the second area) to the total area of ​​the cell, that is, the overlap between the cell and the second area represents the proportion of the area of ​​the cell that falls within the second area. For any cell in the second table, if the overlap between the cell and the second area is less than the second overlap, the electronic device can determine the cell as a target cell and recognize the text in the cell. The second overlap degree may be pre-set, for example, the second overlap degree may be 30%, 40%, etc., which is not limited in the embodiment of the present application.

[0277] In some embodiments, after the electronic device determines the target cells based on the overlap between each cell in the second table and the second region, it can also coordinate-align all the determined target cells in the second table, that is, align all the determined target cells vertically, horizontally, and horizontally in the second table, to obtain a subtable containing all the determined target cells in rows a and columns b from the second table, where a and b are both positive integers. The electronic device can then recognize text in the target cells in the subtable. In this case, the target cells in the identified subtable contain text, while the other cells are empty.

[0278] For example, Figure 22 As shown in Figure (a), the electronic device can determine the coordinates of each cell in the table (i.e., the second table) to which the table content in the second area belongs. The electronic device can then determine the degree of overlap between each cell in the second table and the second area based on the coordinates of each cell in the second table and the coordinate range of the second area. In this case, since the degree of overlap between the second area and the cells other than the cell in the 2nd row and 2nd column, the cell in the 2nd row and 3rd column, and the cell in the 3rd row and 2nd column in the second table is less than 30%, the electronic device determines all cells other than these three cells in the second table as target cells. Afterwards, the electronic device aligns the coordinates of all target cells in the second table to obtain a sub-table with 5 rows and 3 columns from the second table. The electronic device recognizes the text in the target cell in the sub-table, and the recognized sub-table can be as follows Figure 22 As shown in Figure (b), in this case, there is text in the target cell in the identified subtable, while other cells are empty.

[0279] Furthermore, after the electronic device identifies the subtable, it also obtains the encoding information of the subtable. In this case, the electronic device can also extract the identified subtable. Optionally, the electronic device can store the identified subtable (i.e., the encoding information of the subtable) to the system clipboard.

[0280] As an example, if an edit box exists in the screen content of the electronic device, the electronic device may directly paste the sub-table stored in the system clipboard into the edit box.

[0281] As another example, if an edit box exists in the screen content of the electronic device and the edit box obtains focus, the electronic device may paste the sub-table stored in the system clipboard into the edit box.

[0282] As another example, after the electronic device detects a paste instruction in any edit box, it can paste the sub-table stored in the system clipboard into the edit box.

[0283] As another example, after detecting that any edit box has the focus, the electronic device may paste the subtable stored in the system clipboard into the edit box.

[0284] The third case: If the content located in the second area in the screen content of the electronic device is picture content, the electronic device determines that the picture to which the picture content in the second area belongs is the second picture, and identifies other content except the second picture from the screen content of the electronic device according to the coordinates of the second picture. The other content except the second picture in the screen content of the electronic device is the target content.

[0285] The electronic device can first determine the coordinate range of the second area circled by the second mark, and then detect the content in the coordinate range of the second area in the screen content of the electronic device to determine whether the content in the second area is text content, table content, picture content, or formula content. If the content in the second area is picture content, the electronic device can analyze the image features of the picture (i.e., the second picture) to which the picture content in the second area belongs to obtain the coordinates of the second picture. The electronic device can then identify other content other than the second picture from the screen content of the electronic device based on the coordinates of the second picture.

[0286] The fourth case: If the content located in the second area of ​​the screen content of the electronic device is formula content, the electronic device identifies other content in the screen content of the electronic device except the formula (which may be called the second formula) where the formula content in the second area is located, and the other content in the screen content of the electronic device except the second formula is the target content.

[0287] The electronic device may first determine the coordinate range of the second area enclosed by the second marker, and then detect the content within the coordinate range of the second area in the screen content of the electronic device to determine whether the content in the second area is text content, table content, image content, or formula content. If the content in the second area is formula content, the electronic device may directly identify the other content in the screen content of the electronic device except for the formula in the second area.

[0288] The operation of the electronic device identifying the target content from the screen content of the electronic device according to the content blocked by the second mark may include the following four situations:

[0289] The first case: If the content obscured by the second mark in the screen content of the electronic device is text content, the electronic device determines that the text paragraph containing the other text content in the screen content of the electronic device except the text content obscured by the second mark is the second paragraph. The electronic device determines the occlusion degree of each sentence in the second paragraph based on the coordinates of each sentence in the second paragraph and the coordinates of the second mark, and identifies the target sentence from the second paragraph based on the occlusion degree of each sentence in the second paragraph, and the target sentence is the target content; or the electronic device determines the occlusion degree of each character in the second paragraph based on the coordinates of each character in the second paragraph and the coordinates of the second mark, and identifies the target character from the second paragraph based on the occlusion degree of each character in the second paragraph, and the target character is the target content.

[0290] The electronic device can first determine the coordinates of the second mark, and then detect the content located at the coordinates of the second mark in the screen content of the electronic device to determine whether the content blocked by the second mark is text content, table content, picture content, or formula content. If the content blocked by the second mark is text content, the electronic device performs text line detection on the screen content of the electronic device to determine the coordinates of each text line in the screen content of the electronic device, and then performs text paragraph analysis based on the coordinates of each text line in the screen content of the electronic device to obtain the coordinates of each text paragraph in the screen content of the electronic device and the coordinates of each text line in each text paragraph. Afterwards, the electronic device can determine the text paragraph (i.e., the second paragraph) in the screen content of the electronic device where other text content other than the text content blocked by the second mark is located based on the coordinates of the second mark, the coordinates of each text paragraph in the screen content of the electronic device, and the coordinates of each text line in each text paragraph. At this time, the coordinates of the second paragraph and the coordinates of each text line in the second paragraph are also determined.

[0291] As an example, the electronic device can determine the coordinates of each sentence in the second paragraph. For example, the electronic device can use the segmentation algorithm in NLP to determine the coordinates of each sentence in the second paragraph. The electronic device can then determine the occlusion degree of each sentence in the second paragraph based on the coordinates of each sentence in the second paragraph and the coordinates of the second marker. Optionally, the occlusion degree of a sentence in the second paragraph can be the ratio of the occlusion area of ​​the sentence (that is, the area of ​​the sentence blocked by the second marker) to the total area of ​​the sentence, that is, the occlusion degree of the sentence represents the proportion of the area of ​​the sentence blocked by the second marker. For any sentence in the second paragraph, if the occlusion degree of the sentence is less than the first occlusion degree, the electronic device can determine the sentence as the target sentence and identify the sentence. Among them, the first occlusion degree can be set in advance. For example, the first occlusion degree can be 30%, 40%, etc., which is not limited in the embodiments of the present application.

[0292] As another example, the electronic device can determine the coordinates of each character in the second paragraph. For example, the character may include letters, numbers, Chinese characters, symbols, etc. The electronic device can then determine the occlusion degree of each character in the second paragraph based on the coordinates of each character in the second paragraph and the coordinates of the second mark. Optionally, the occlusion degree of a character in the second paragraph can be the ratio of the occlusion area of ​​the character (i.e., the area of ​​the character obscured by the second mark) to the total area of ​​the character, that is, the occlusion degree of the character represents the proportion of the area of ​​the character that is obscured by the second mark. For any character in the second paragraph, if the occlusion degree of the character is less than the first occlusion degree, the electronic device can determine the character as the target character and recognize the character.

[0293] Furthermore, after the electronic device recognizes the target sentence from the second paragraph, it also obtains the encoding information of the target sentence. In this case, the electronic device can also extract the recognized target sentence. Optionally, the electronic device can store the recognized target sentence (i.e., the encoding information of the target sentence) to the system clipboard.

[0294] Similarly, after the electronic device recognizes the target character from the second paragraph, it also obtains the encoding information of the target character. In this case, the electronic device can also extract the recognized target character. Optionally, the electronic device can store the recognized target character (i.e., the encoding information of the target character) to the system clipboard.

[0295] As an example, if an edit box exists in the screen content of the electronic device, the electronic device can directly paste the target content (target sentence or target character) stored in the system clipboard into the edit box.

[0296] As another example, if an edit box exists in the screen content of the electronic device and the edit box obtains focus, the electronic device may paste the target content stored in the system clipboard into the edit box.

[0297] As another example, after the electronic device detects a paste instruction in any edit box, it can paste the target content stored in the system clipboard into the edit box.

[0298] As another example, after detecting that any edit box has obtained focus, the electronic device may paste the target content stored in the system clipboard into the edit box.

[0299] In the second scenario, if the content obscured by the second marker on the screen of the electronic device is table content, the electronic device determines that the table to which the table content obscured by the second marker belongs is the second table. The electronic device determines the obscuration degree of each cell in the second table based on the coordinates of each cell in the second table and the coordinates of the second marker; and identifies text in a target cell from the second table based on the obscuration degree of each cell in the second table, where the text in the target cell is the target content.

[0300] The electronic device can first determine the coordinates of the second marker, and then detect the content at the coordinates of the second marker in the screen content of the electronic device to determine whether the content blocked by the second marker is text content, table content, picture content, or formula content. If the content blocked by the second marker is table content, the electronic device analyzes the table structure of the table (i.e., the second table) to which the table content blocked by the second marker belongs to obtain the coordinates of each cell in the second table. The electronic device can then determine the occlusion degree of each cell in the second table based on the coordinates of each cell in the second table and the coordinates of the second marker. Optionally, the occlusion degree of a cell in the second table can be the ratio of the occlusion area of ​​the cell (i.e., the area of ​​the cell blocked by the second marker) to the total area of ​​the cell, that is, the occlusion degree of the cell represents the proportion of the area of ​​the cell blocked by the second marker. For any cell in the second table, if the occlusion degree of the cell is less than the second occlusion degree, the electronic device can determine the cell as a target cell and recognize the text in the cell. The second overlap degree may be pre-set, for example, the second overlap degree may be 10%, 20%, etc., which is not limited in the embodiment of the present application.

[0301] In some embodiments, after the electronic device determines the target cells based on the occlusion levels of each cell in the second table, it can also coordinate-align all the determined target cells in the second table. That is, all the determined target cells are aligned vertically, horizontally, and left-right in the second table to obtain a subtable containing all the determined target cells in rows a and columns b from the second table, where a and b are both positive integers. The electronic device can then recognize text in the target cells in the subtable. In this case, the target cells in the identified subtable contain text, while the other cells are empty.

[0302] Furthermore, after the electronic device identifies the subtable, it also obtains the encoding information of the subtable. In this case, the electronic device can also extract the identified subtable. Optionally, the electronic device can store the identified subtable (i.e., the encoding information of the subtable) to the system clipboard.

[0303] As an example, if an edit box exists in the screen content of the electronic device, the electronic device may directly paste the sub-table stored in the system clipboard into the edit box.

[0304] As another example, if an edit box exists in the screen content of the electronic device and the edit box obtains focus, the electronic device may paste the sub-table stored in the system clipboard into the edit box.

[0305] As another example, after the electronic device detects a paste instruction in any edit box, it can paste the sub-table stored in the system clipboard into the edit box.

[0306] As another example, after detecting that any edit box has the focus, the electronic device may paste the subtable stored in the system clipboard into the edit box.

[0307] The third case: If the content in the screen content of the electronic device that is blocked by the second mark is picture content, the electronic device determines that the picture to which the picture content blocked by the second mark belongs is the second picture, and identifies other content except the second picture from the screen content of the electronic device based on the coordinates of the second picture. The other content in the screen content of the electronic device except the second picture is the target content.

[0308] The electronic device can first determine the coordinates of the second mark, and then detect the content located at the coordinates of the second mark in the screen content of the electronic device to determine whether the content blocked by the second mark is text content, table content, picture content, or formula content. If the content blocked by the second mark is picture content, the electronic device can analyze the image features of the picture (i.e., the second picture) to which the picture content blocked by the second mark belongs to obtain the coordinates of the second picture. The electronic device can then identify other content other than the second picture from the screen content of the electronic device based on the coordinates of the second picture.

[0309] The fourth case: If the content in the screen content of the electronic device that is blocked by the second mark is formula content, the electronic device identifies other content in the screen content of the electronic device except the formula (which may be called the second formula) in which the formula content blocked by the second mark is located, and the other content in the screen content of the electronic device except the second formula is the target content.

[0310] The electronic device may first determine the coordinates of the second marker, and then detect the content located at the coordinates of the second marker in the electronic device's screen content to determine whether the content obscured by the second marker is text content, table content, image content, or formula content. If the content obscured by the second marker is a formula content, the electronic device may directly identify the other content in the electronic device's screen content except for the formula content obscured by the second marker.

[0311] The third method: if the designated mark includes a first mark and a second mark, the electronic device may identify the target content from the screen content of the electronic device through the following methods 1 to 6.

[0312] Method 1: If the first marker and the second marker do not overlap, the electronic device may retain the first marker in the designated marker and discard the second marker, and then identify the target content from the electronic device's screen content using the first method described above. Alternatively, if the first marker and the second marker do not overlap, the electronic device may discard the first marker and retain the second marker in the designated marker, and then identify the target content from the electronic device's screen content using the second method described above.

[0313] In the embodiment of the present application, if both the first mark and the second mark exist but do not overlap, then the delineation intentions of the two marks conflict. In this case, one of the two marks can be discarded and the other can be retained so that content recognition can continue normally.

[0314] For example, Figure 23 As shown, the electronic device obtains the specified tag, assuming that the specified tag is Figure 24 The electronic device then determines the overlap between the first and second markers. Figure 24 The first mark and the second mark shown in Figure (a) do not overlap, so the electronic device can determine that the overlap of the first mark and the second mark is: the first mark and the second mark do not overlap. In this case, the electronic device can retain the first mark and discard the second mark, and obtain Figure 24 The designated mark shown in Figure (b) in the figure includes the first mark but does not include the second mark, so the target content can be identified from the screen content of the electronic device by the first method mentioned above. Alternatively, the electronic device can discard the first mark and retain the second mark, and obtain Figure 24 The designated mark shown in Figure (c) in the figure, at this time, the designated mark includes the second mark but does not include the first mark, so the target content can be identified from the screen content of the electronic device through the second method mentioned above.

[0315] Method 2: If the second mark is located within the first mark and the second mark is used to circle content that does not need to be identified, the electronic device can use the other areas in the first area circled by the first mark except the second area circled by the second mark as the third area, and then identify the target content from the screen content of the electronic device based on the third area.

[0316] The third area is used to indicate the content to be identified. The operation of the electronic device identifying the target content from the screen content of the electronic device based on the third area is similar to the operation of the electronic device identifying the target content from the screen content of the electronic device based on the first area in the first embodiment described above, and this embodiment of the application will not be repeated.

[0317] For example, if the content in the third area of ​​the screen content of the electronic device is text content, the electronic device determines that the text paragraph in which the text content in the third area is located is the third paragraph. The electronic device determines the degree of overlap between each sentence in the third paragraph and the third area based on the coordinates of each sentence in the third paragraph and the coordinate range of the third area, and identifies a target sentence from the third paragraph based on the degree of overlap between each sentence in the third paragraph and the third area, where the target sentence is the target content. Alternatively, the electronic device determines the degree of overlap between each character in the third paragraph and the third area based on the coordinates of each character in the third paragraph and the coordinate range of the third area, and identifies a target character from the third paragraph based on the degree of overlap between each character in the third paragraph and the third area, where the target character is the target content.

[0318] If the content in the third area of ​​the screen content of the electronic device is table content, the electronic device determines that the table to which the table content in the third area belongs is the third table. The electronic device determines the degree of overlap between each cell in the third table and the third area based on the coordinates of each cell in the third table and the coordinate range of the third area; and identifies text in a target cell in the third table based on the degree of overlap between each cell in the third table and the third area, where the text in the target cell is the target content.

[0319] If the content in the third area of ​​the screen content of the electronic device is picture content, the electronic device determines that the picture to which the picture content in the third area belongs is the third picture, identifies the third picture according to the coordinates of the third picture, and the third picture is the target content.

[0320] If the content in the third area of ​​the screen content of the electronic device is formula content, the electronic device identifies the formula (which may be referred to as the third formula) in which the formula content in the third area is located, and the third formula is the target content.

[0321] In some embodiments, if the second mark is located within the first mark and the second mark is used to circle content that does not need to be identified, the electronic device can identify the target content from the screen content of the electronic device through the above-mentioned method 2. Alternatively, the electronic device can retain the first mark and discard the second mark in the specified mark, and then identify the target content from the screen content of the electronic device through the above-mentioned first method. Alternatively, the electronic device can discard the first mark and retain the second mark in the specified mark, and then identify the target content from the screen content of the electronic device through the above-mentioned second method.

[0322] For example, Figure 25 As shown, the electronic device obtains the specified tag, assuming that the specified tag is Figure 26 The electronic device then determines the overlap between the first and second markers. Figure 26 The second mark shown in Figure (a) is inside the first mark, so the electronic device can determine that the overlap of the first mark and the second mark is: the second mark is inside the first mark. In this case, the electronic device can retain the first mark and the second mark, and obtain Figure 26 The designated mark shown in Figure (b) in the figure can be used to identify the target content from the screen content of the electronic device through the above method 2. Alternatively, the electronic device can retain the first mark and discard the second mark to obtain Figure 26 The designated mark shown in Figure (c) in the figure includes the first mark but does not include the second mark, so the target content can be identified from the screen content of the electronic device by the first method mentioned above. Alternatively, the electronic device can discard the first mark and retain the second mark, and obtain Figure 26 The designated mark shown in Figure (d) in the figure, at this time, the designated mark includes the second mark but does not include the first mark, so the target content can be identified from the screen content of the electronic device through the second method mentioned above.

[0323] Method 3: If the second mark is located within the first mark and the second mark is used to block content that does not need to be identified, the electronic device identifies the target content from the screen content of the electronic device based on the first area circled by the first mark, and then deletes the content blocked by the second mark from the identified target content.

[0324] The operation of the electronic device identifying the target content from the screen content of the electronic device according to the first area circled by the first mark is similar to the operation of the electronic device identifying the target content from the screen content of the electronic device according to the first area circled by the first mark in the first method mentioned above, and the embodiments of the present application will not be repeated here.

[0325] The operation of the electronic device deleting the content obscured by the second mark from the identified target content may be: if the identified target content is text, the electronic device determines the occlusion degree of each sentence in the target content based on the coordinates of each sentence in the target content and the coordinates of the second mark, and if the occlusion degree of a sentence in the target content is greater than or equal to the first occlusion degree, the sentence is deleted from the target content; or, the electronic device determines the occlusion degree of each character in the target content based on the coordinates of each character in the target content and the coordinates of the second mark, and if the occlusion degree of a character in the target content is greater than or equal to the first occlusion degree, the character is deleted from the target content.

[0326] If the identified target content is a table, the electronic device determines the occlusion degree of each cell in the target content based on the coordinates of each cell in the target content and the coordinates of the second mark. If the occlusion degree of a cell in the target content is greater than or equal to the second occlusion degree, the text in this cell is deleted from the target content.

[0327] In particular, if the identified target content is a picture or a formula, the electronic device discards the second mark, that is, the electronic device does not need to delete the content blocked by the second mark from the identified target content.

[0328] In some embodiments, if the second mark is located within the first mark and the second mark is used to block content that does not need to be identified, the electronic device can identify the target content from the screen content of the electronic device through the above-mentioned method 3. Alternatively, the electronic device can retain the first mark and discard the second mark in the specified mark, and then identify the target content from the screen content of the electronic device through the above-mentioned first method. Alternatively, the electronic device can discard the first mark and retain the second mark in the specified mark, and then identify the target content from the screen content of the electronic device through the above-mentioned second method.

[0329] Method 4: If the first mark is located within the second mark, and the second mark is used to circle content that does not need to be identified, the electronic device identifies the target content from the screen content of the electronic device based on the first area circled by the first mark, and identifies the target content from the screen content of the electronic device based on the second area circled by the second mark.

[0330] The operation of the electronic device identifying the target content from the screen content of the electronic device according to the first area circled by the first mark is similar to the operation of the electronic device identifying the target content from the screen content of the electronic device according to the first area circled by the first mark in the first method mentioned above, and the embodiments of the present application will not be repeated here.

[0331] The operation of the electronic device identifying the target content from the screen content of the electronic device according to the second area circled by the second mark is similar to the operation of the electronic device identifying the target content from the screen content of the electronic device according to the second area circled by the second mark in the above-mentioned second method, and the embodiments of the present application will not be repeated here.

[0332] In some embodiments, if the first mark is located within the second mark and the second mark is used to circle content that does not need to be identified, the electronic device can identify the target content from the screen content of the electronic device using the above-mentioned method 4. Alternatively, the electronic device can retain the first mark and discard the second mark in the specified mark, and then identify the target content from the screen content of the electronic device using the above-mentioned first method. Alternatively, the electronic device can discard the first mark and retain the second mark in the specified mark, and then identify the target content from the screen content of the electronic device using the above-mentioned second method.

[0333] For example, Figure 27 As shown, the electronic device obtains the specified tag, assuming that the specified tag is Figure 28 The electronic device then determines the overlap between the first and second markers. Figure 28 The first mark shown in Figure (a) is inside the second mark, so the electronic device can determine that the overlap of the first mark and the second mark is: the first mark is inside the second mark. In this case, the electronic device can retain the first mark and the second mark, and obtain Figure 28 The designated mark shown in FIG (b) in FIG, at this time, the target content can be identified from the screen content of the electronic device by the above method 4. Alternatively, the electronic device can retain the first mark and discard the second mark, and obtain Figure 28 The designated mark shown in Figure (c) in the figure includes the first mark but does not include the second mark, so the target content can be identified from the screen content of the electronic device by the first method mentioned above. Alternatively, the electronic device can discard the first mark and retain the second mark, and obtain Figure 28 The designated mark shown in Figure (d) in the figure, at this time, the designated mark includes the second mark but does not include the first mark, so the target content can be identified from the screen content of the electronic device through the second method mentioned above.

[0334] Method 5: If the first marker intersects the second marker and the second marker is used to circle content that does not need to be recognized, the electronic device retains the first marker and uses the intersection of the first and second markers as the new second marker, or retains the second marker and uses the union of the second marker and the first marker as the new first marker. The target content is then recognized from the screen content of the electronic device using Method 2 above.

[0335] For example, Figure 29 As shown, the electronic device obtains the specified tag, assuming that the specified tag is Figure 30 The electronic device then determines the overlap between the first and second markers. Figure 30 The first mark shown in Figure (a) intersects with the second mark, so the electronic device can determine that the overlap of the first mark and the second mark is: the first mark intersects with the second mark. In this case, the electronic device can retain the first mark and use the intersection of the first mark and the second mark as the new second mark, and obtain Figure 30 The designated mark shown in Figure (b) in the figure is shown in FIG. 2. At this time, the second mark is located within the first mark, so the target content can be identified from the screen content of the electronic device through the above method 2. Alternatively, the electronic device can retain the second mark and use the union of the second mark and the first mark as the new first mark to obtain Figure 30As shown in FIG. 3(c), the second mark is located within the first mark, and thus the target content can be identified from the screen content of the electronic device through the above-mentioned method 2.

[0336] Method 6: If the first mark intersects the second mark and the second mark is used to block content that does not need to be identified, the electronic device retains the first mark and uses the part of the second mark located within the first mark as the new second mark, and then identifies the target content from the screen content of the electronic device through the above method 3.

[0337] For example, Figure 31 As shown, the electronic device obtains the specified tag, assuming that the specified tag is Figure 32 The electronic device then determines the overlap between the first and second markers. Figure 32 The first mark shown in Figure (a) of the figure intersects the second mark, so the electronic device can determine that the overlap of the first mark and the second mark is: the first mark intersects the second mark. In this case, the electronic device can retain the first mark and use the part of the second mark located within the first mark as the new second mark, and obtain Figure 32 As shown in FIG. 3 (b), the second mark is located within the first mark, and thus the target content can be identified from the screen content of the electronic device through the above-mentioned method 3.

[0338] In some embodiments, if the screen content of the electronic device is a video playback interface, the screen content recognition method provided in the embodiments of the present application can be used to recognize the video image of a video in a paused state in the video playback interface. In this case, after the user draws a designated mark in the video image in the video playback interface, they can also choose to apply the drawn designated mark in real time during video playback.

[0339] For example, the electronic device may provide a designated button. By operating the designated button, the user may choose to continuously apply the designated mark during video playback. In this case, during video playback, the electronic device may identify the target content from each frame of the video image played based on the designated mark. Alternatively, by operating the designated button, the user may choose to continuously apply the designated mark when the video playback time point is within certain time periods. In this case, when the video playback reaches a point within these time periods, the electronic device may identify the target content from each frame of the video image played based on the designated mark.

[0340] Optionally, after the electronic device recognizes the target content during video playback, it can store the recognized target content in the system clipboard when the video playback ends. Afterwards, the electronic device can automatically paste the content in the system clipboard into the edit box, or the user can manually paste the content in the system clipboard into the edit box.

[0341] Optionally, after the electronic device recognizes the target content during video playback, it can also store the recognized target content in a designated file when the video playback ends, so that the user can view the recognized target content in the designated file after the video playback ends.

[0342] For example, Figure 33 As shown, the electronic device receives the video code stream and then renders the video code stream to a visual interface for video playback. Assume that the visual interface is Figure 34 In this case, if the user wants to obtain the subtitle information of the video played in the video playing interface 3401, the user can pause a certain frame of the video image and then draw a specified mark on the screen, for example, Figure 34 As shown in FIG. 34 (a), the user can draw a first mark in the subtitle area of ​​the video playback interface 3401, or, as shown in FIG. Figure 34 As shown in FIG. 34(b), the user can draw a second mark in the non-subtitle area of ​​the video playback interface 3401. In this way, the electronic device can identify the target content from each frame of the video image played according to the specified mark during the video playback process. For example, Figure 34 In this case, the electronic device can output the recognized subtitle information after the video is played, so that the user can easily and conveniently obtain the subtitle information of the entire video, thereby improving the user experience.

[0343] For ease of understanding, the following Figure 35 The above-mentioned screen content recognition method is exemplified.

[0344] Figure 35 Schematic diagram of a screen content recognition method provided by an embodiment of the present application. Figure 35 The method comprises the following steps (1) to (8):

[0345] (1) The user draws a trajectory line on the screen of the electronic device.

[0346] (2) The electronic device obtains a designated mark (ie, the first mark and / or the second mark) according to the drawn trajectory line.

[0347] (3) If the designated mark includes only the first mark or the second mark, the electronic device does not process the designated mark. If the designated mark includes the first mark and the second mark, and the first mark and the second mark intersect, the electronic device will process the designated mark according to the above. Figure 12 In the third method of step 1203 in the embodiment, the designated mark is processed in the corresponding manner 5 or 6. If the designated mark includes the first mark and the second mark, and the second mark is located within the first mark, then according to the above Figure 12 In the third method of step 1203 in the embodiment, the designated mark is processed in the second method or the third method. If the designated mark includes the first mark and the second mark, and the first mark is located within the second mark, the electronic device processes the designated mark according to the above method. Figure 12 In the third method of step 1203 in the embodiment, the designated mark is processed accordingly in method 4. If the designated mark includes the first mark and the second mark, and the first mark and the second mark do not overlap, then the designated mark is processed accordingly in accordance with the above method. Figure 12 In the third method in step 1203 of the embodiment, method 1 performs corresponding processing on the designated mark. Afterwards, the electronic device performs subsequent steps according to the designated mark to identify the target content from the screen content of the electronic device.

[0348] (4) The electronic device performs text line detection, component area detection, and component type detection on the screen content to determine the text area, table area, image area, and formula area in the screen content.

[0349] Components may include tables, pictures, and formulas. To determine the text area, after performing text line detection, text paragraph analysis may be performed to determine the coordinates of each text paragraph and the coordinates of each text line within each text paragraph.

[0350] It is worth noting that in the embodiment of the present application, lightweight detection can be achieved through text line detection, component area detection, and component type detection. Since this detection does not identify specific content, it can avoid the situation where the entire article is slowly recognized.

[0351] (5) If the content indicated by the designated mark is in the text area, the electronic device Figure 12 In the first case of step 1203 in the embodiment, the target content is identified. If the content indicated by the designated mark is in the table area, the electronic device Figure 12 The second case in step 1203 of the embodiment is used to identify the target content. If the content indicated by the designated mark is in the picture area, the electronic device Figure 12 The third case in step 1203 of the embodiment is used to identify the target content. If the content indicated by the designated mark is in the formula area, the electronic device Figure 12The fourth case in step 1203 in the embodiment is used to identify the target content.

[0352] (6) If the identified target content has multiple types, such as at least two of text, table, picture, and formula, the electronic device can sort the multiple target contents according to the corresponding types.

[0353] (7) The electronic device stores the target content to the system clipboard.

[0354] (8) The electronic device pastes the target content in the system clipboard into the edit box.

[0355] The following combination Figure 36 To the above Figure 35 The embodiments are described in detail with examples.

[0356] Figure 36 This is a flow chart of a method for identifying screen content provided by an embodiment of the present application. Figure 36 The method includes the following steps 3601 to 3610:

[0357] Step 3601: The user draws a trajectory line on the screen, and the designated mark acquisition module acquires the drawn trajectory line.

[0358] Step 3602: The designated mark acquisition module acquires the designated mark according to the drawn trajectory line.

[0359] Step 3603: The designated mark processing module processes the designated mark according to the overlap between the first mark and the second mark in the designated mark. For specific operations, please refer to the above. Figure 35 Step (3) in the embodiment.

[0360] Step 3604: If the content indicated by the designated mark is text content, the content recognition module performs text line detection and then paragraph analysis to obtain the coordinates of each text paragraph and the coordinates of each text line within each text paragraph.

[0361] Step 3605: The content identification module determines the candidate paragraphs based on the coordinate range of the text content indicated by the designated mark and the coordinates of each text paragraph. For specific operations, please refer to the above. Figure 12 The first case in step 1203 in the embodiment.

[0362] The coordinate range of the text content indicated by the designated mark is the coordinate range of the first area circled by the first mark, and / or the coordinate range of the second area circled by the second mark or the coordinates of the second mark.

[0363] The candidate paragraph is a text paragraph containing text content in the first area circled by the first mark, and / or the candidate paragraph is a text paragraph containing text content other than text content in the second area circled by the second mark or text content obscured by the second mark.

[0364] As an example, the target sentence may be recognized through the following steps 3606 and 3607. As another example, the target character may be recognized through the following step 3608.

[0365] Step 3606: The content identification module obtains the coordinates of each sentence in the candidate paragraph through the NLP module.

[0366] Step 3607: The content identification module identifies the target sentence based on the coordinate range of the text content indicated by the designated mark and the coordinates of each sentence in the candidate paragraph.

[0367] Step 3608: The content recognition module recognizes the target character according to the coordinate range of the text content indicated by the designated mark and the coordinates of each character in the candidate paragraph.

[0368] Step 3609: The content recognition module stores the recognized content to the system clipboard.

[0369] Step 3610: Paste the content in the system clipboard into the edit box when the edit box has focus.

[0370] The following combination Figure 37 The content recognition processing procedure when the content indicated by the designation mark is text content will be described.

[0371] See also Figure 37 After text line detection and text paragraph analysis, the coordinates of the text paragraph and the coordinates of the text line within the text paragraph are obtained. After that, the candidate paragraph is determined based on the specified mark, the coordinates of the text paragraph and the coordinates of the text line within the text paragraph. For specific operations, please refer to the above Figure 36 Step 3605 in the embodiment. Thereafter, it may be determined whether the designated tag is the first tag or the second tag.

[0372] If the designated mark is the first mark, text recognition is performed on the candidate paragraph. After that, sentence coordinate detection is performed on the candidate paragraph to obtain the coordinates of each sentence in the candidate paragraph, and the overlap of each sentence is determined based on the coordinates of each sentence in the candidate paragraph. If the overlap of a sentence is high, the sentence is extracted; if the overlap of a sentence is low, the sentence is not extracted. Alternatively, character coordinate detection is performed on the candidate paragraph to obtain the coordinates of each character in the candidate paragraph, and the overlap of each character is determined based on the coordinates of each character in the candidate paragraph. If the overlap of a character is high, the character is extracted; if the overlap of a character is low, the character is not extracted.

[0373] If the designated mark is the second mark, text recognition is performed on the candidate paragraph. Afterwards, sentence coordinate detection is performed on the candidate paragraph to obtain the coordinates of each sentence in the candidate paragraph, and the overlap of each sentence is determined based on the coordinates of each sentence in the candidate paragraph. If the overlap of a sentence is high, the sentence is not extracted; if the overlap of a sentence is low, the sentence is extracted. Alternatively, character coordinate detection is performed on the candidate paragraph to obtain the coordinates of each character in the candidate paragraph, and the overlap of each character is determined based on the coordinates of each character in the candidate paragraph. If the overlap of a character is high, the character is not extracted; if the overlap of a character is low, the character is extracted.

[0374] The following combination Figure 38 The following describes the content recognition process when the content indicated by the designation mark is table content.

[0375] See also Figure 38 After obtaining the specified tag, if the content indicated by the specified tag is in the table area, the table structure detection is performed. Then, it can be determined whether the specified tag is the first tag or the second tag.

[0376] If the specified tag is the first tag, the target cell is determined based on the degree of cell overlap in the table. The target cell is aligned vertically, horizontally, and left-right to obtain a subtable with a row and b columns. Text recognition is performed on the subtable, and the subtable is then output.

[0377] If the specified flag is the second flag, text recognition is performed on the entire table. The target cell is determined based on the degree of cell overlap. The contents of all cells in the table except the target cell are cleared, and the entire table is output.

[0378] The following combination Figure 39 The content recognition processing process when the content indicated by the designated tag is image content will be described.

[0379] See also Figure 39 After obtaining the specified tag, if the content indicated by the specified tag is in the image area, the specified tag is determined to be the first tag or the second tag. If the specified tag is the second tag, the image is not recognized. If the specified tag is the first tag, the image is recognized and corrected, and the recognized image is output.

[0380] The following combination Figure 40 The content recognition processing process when the content indicated by the designated tag is image content will be described.

[0381] See also Figure 40After obtaining the specified tag, if the content indicated by the specified tag is in the formula area, the specified tag is determined to be the first tag or the second tag. If the specified tag is the second tag, the formula is not recognized. If the specified tag is the first tag, the formula is recognized and output.

[0382] Figure 41 This is a schematic diagram of the structure of a screen content recognition device provided by an embodiment of the present application. The device can be implemented as part or all of a computer device by software, hardware, or a combination of both. The computer device can be Figures 1 to 2 The electronic device 100 according to the embodiment. Figure 41 The device includes: a first acquisition module 4101, a second acquisition module 4102 and an identification module 4103.

[0383] The first acquisition module 4101 is used to execute the above Figure 12 Step 1201 in the embodiment;

[0384] The second acquisition module 4102 is used to execute the above Figure 12 Step 1202 in the embodiment;

[0385] Identification module 4103, used to execute the above Figure 12 Step 1203 in the embodiment.

[0386] Optionally, the screen content is an image or interface displayed by the electronic device, and the interface is an application interface, a video playback interface, or a camera preview interface.

[0387] Optionally, the first mark is a closed figure and the second mark is an open figure; or, the first mark is a first preset figure and the second mark is a second preset figure, and the shapes of the first preset figure and the second preset figure are different; or, the first mark is a closed figure and the second mark is a closed figure, and the line thickness of the first mark and the second mark is different; or, the first mark is a closed figure and the second mark is a closed figure, and the line color of the first mark and the second mark is different.

[0388] Optionally, the first mark is a closed figure, and the second mark is a non-closed figure, and the second acquisition module 4102 is used to:

[0389] Perform polygon fitting on n trajectory lines;

[0390] If a polygon is fitted by at least one of the n trajectory lines, the fitted polygon is determined as a first mark, and the trajectory line of the n trajectory lines that does not fit a polygon is determined as a second mark;

[0391] If multiple polygons are fitted by at least one of the n trajectory lines, a first mark is determined according to the multiple fitted polygons and the overlap of the multiple polygons, and a trajectory line of the n trajectory lines that does not fit a polygon is determined as a second mark;

[0392] If no polygon is fitted through the n trajectory lines, the n trajectory lines are determined as second markers.

[0393] Optionally, the first mark is used to circle content that needs to be identified, and the second mark is used to circle content that does not need to be identified or to cover content that does not need to be identified. The identification module 4103 is used to:

[0394] If the designated mark includes the first mark and does not include the second mark, identifying target content from the screen content of the electronic device according to the first area circled by the first mark;

[0395] If the designated mark includes the second mark but does not include the first mark, the target content is identified from the screen content of the electronic device according to the second area circled by the second mark or according to the content blocked by the second mark.

[0396] Optionally, the identification module 4103 is used to:

[0397] If the content in the first area of ​​the screen content of the electronic device is text content, determining that the text paragraph in which the text content in the first area is located is the first paragraph;

[0398] Based on the coordinates of each sentence in the first paragraph and the coordinate range of the first area, the overlap between each sentence in the first paragraph and the first area is determined, and based on the overlap between each sentence in the first paragraph and the first area, a target sentence is identified from the first paragraph, where the target sentence is the target content; or, based on the coordinates of each character in the first paragraph and the coordinate range of the first area, the overlap between each character in the first paragraph and the first area is determined, and based on the overlap between each character in the first paragraph and the first area, a target character is identified from the first paragraph, where the target character is the target content.

[0399] Optionally, the identification module 4103 is used to:

[0400] If the content in the first area of ​​the screen content of the electronic device is table content, determining that the table to which the table content in the first area belongs is a first table;

[0401] Determining the degree of overlap between each cell in the first table and the first region based on the coordinates of each cell in the first table and the coordinate range of the first region;

[0402] The text in the target cell is identified from the first table according to the overlap between each cell in the first table and the first region, and the text in the target cell is the target content.

[0403] Optionally, the identification module 4103 is used to:

[0404] If the content in the first area of ​​the screen content of the electronic device is picture content, determining that the picture to which the picture content in the first area belongs is the first picture;

[0405] The first picture is identified according to the coordinates of the first picture, where the first picture is the target content.

[0406] Optionally, the identification module 4103 is used to:

[0407] If the content in the first area of ​​the screen content of the electronic device is formula content, the formula where the formula content in the first area is located is identified, and the formula where the formula content in the first area is located is the target content.

[0408] Optionally, the first mark is used to circle content that needs to be identified, and the second mark is used to circle content that does not need to be identified or to cover content that does not need to be identified. The identification module 4103 is used to:

[0409] If the designated mark includes a first mark and a second mark, and the first mark and the second mark do not overlap, then the first mark is retained in the designated mark and the second mark is discarded, and the target content is identified from the screen content of the electronic device according to the first area circled by the first mark; or,

[0410] If the designated mark includes a first mark and a second mark, and the first mark and the second mark do not overlap, the first mark is discarded in the designated mark and the second mark is retained, and the target content is identified from the screen content of the electronic device based on the second area circled by the second mark or the content blocked by the second mark.

[0411] Optionally, the first mark is used to circle content that needs to be identified, and the second mark is used to circle content that does not need to be identified. The identification module 4103 is used to:

[0412] If the designated mark includes a first mark and a second mark, and the second mark is located within the first mark, then the other areas in the first area circled by the first mark except the second area circled by the second mark are determined as the third area, and the target content is identified from the screen content of the electronic device based on the third area.

[0413] Optionally, the first mark is used to circle the content that needs to be identified, and the second mark is used to block the content that does not need to be identified. The identification module 4103 is used to:

[0414] If the designated mark includes a first mark and a second mark, and the second mark is located within the first mark, target content is identified from the screen content of the electronic device according to the first area circled by the first mark, and content obscured by the second mark is deleted from the identified target content.

[0415] Optionally, the first mark is used to circle content that needs to be identified, and the second mark is used to circle content that does not need to be identified. The identification module 4103 is used to:

[0416] If the designated tag includes a first tag and a second tag, and the first tag intersects the second tag, then the first tag is retained and the intersection of the first tag and the second tag is used as the new second tag, or the second tag is retained and the union of the second tag and the first tag is used as the new first tag;

[0417] The rest of the first area circled by the first mark except the second area circled by the second mark is determined as a third area, and target content is identified from the screen content of the electronic device according to the third area.

[0418] Optionally, the first mark is used to circle the content that needs to be identified, and the second mark is used to block the content that does not need to be identified. The identification module 4103 is used to:

[0419] If the designated mark includes a first mark and a second mark, and the first mark intersects the second mark, the first mark is retained, and the portion of the second mark located within the first mark is used as a new second mark;

[0420] Target content is identified from the screen content of the electronic device according to the first area circled by the first mark, and content obscured by the second mark is deleted from the identified target content.

[0421] Optionally, the device further comprises:

[0422] A storage module, used to store the identified target content to the system clipboard;

[0423] The paste module is used to paste the target content stored in the system clipboard into the edit box if it is detected that any edit box has obtained the focus.

[0424] In the embodiment of the present application, it is not necessary to recognize the entire screen content of the electronic device. Instead, the target content that meets the user's needs can be identified from the screen content of the electronic device based on the designated mark drawn by the user. The recognition time is short, the content recognition can be completed quickly, and power consumption can be saved. In addition, since the user can choose the content to be recognized or not recognized according to their own needs, that is, the user can select the content in a local area of ​​the screen content to be recognized or not, or can select the content in multiple discontinuous areas of the screen content to be recognized or not, the present application can realize cross-line recognition of screen content, thereby making the recognition of screen content more flexible.

[0425] It should be noted that: when the screen content recognition device provided in the above embodiment recognizes the screen content, it only uses the division of the above-mentioned functional modules as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0426] The functional units and modules in the above embodiments may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The above integrated units may be implemented in the form of hardware or software functional units. In addition, the specific names of the functional units and modules are only for the purpose of distinguishing them from each other and are not intended to limit the scope of protection of the embodiments of this application.

[0427] The screen content recognition device and screen content recognition method provided in the above embodiments belong to the same concept. The specific working process and technical effects brought about by the units and modules in the above embodiments can be found in the method embodiment part and will not be repeated here.

[0428] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (such as a coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0429] The above are optional embodiments provided for this application and are not intended to limit this application. Any modifications, equivalent replacements, improvements, etc. made within the technical scope disclosed in this application should be included in the scope of protection of this application.

Claims

1. A screen content recognition method, characterized in that: Applied to electronic equipment, the method includes: If a screen content recognition instruction is received, obtaining n trajectory lines drawn on the screen of the electronic device, where n is a positive integer; Acquire designated marks according to the n trajectory lines, the designated marks including a first mark and / or a second mark, the first mark being used to circle content to be identified, and the second mark being used to circle content not to be identified or to cover content not to be identified; identifying target content from screen content of the electronic device according to the designated mark; The step of identifying target content from the screen content of the electronic device according to the designated mark includes: If the designated mark includes the first mark and does not include the second mark, identifying the target content from the screen content of the electronic device according to the first area circled by the first mark; If the designated mark includes the second mark but does not include the first mark, the target content is identified from the screen content of the electronic device based on the second area circled by the second mark or based on the content blocked by the second mark.

2. The method according to claim 1, wherein The screen content is an image or interface displayed by the electronic device, and the interface is an application interface, a video playback interface, or a camera preview interface.

3. The method according to claim 1, wherein The first mark is a closed figure, and the second mark is a non-closed figure; or, The first mark is a first preset graphic, the second mark is a second preset graphic, and the first preset graphic and the second preset graphic have different shapes; or, The first mark is a closed figure, the second mark is a closed figure, and the first mark and the second mark have different line thicknesses; or, The first mark is a closed figure, the second mark is a closed figure, and the line colors of the first mark and the second mark are different.

4. The method according to claim 1, wherein The first mark is a closed figure, the second mark is a non-closed figure, and obtaining the designated mark according to the n trajectory lines includes: Performing polygon fitting processing on the n trajectory lines; If a polygon is fitted by at least one of the n trajectory lines, the fitted polygon is determined as the first mark, and the trajectory line of the n trajectory lines that does not fit a polygon is determined as the second mark; If multiple polygons are fitted by at least one of the n trajectory lines, the first mark is determined according to the overlap between the multiple fitted polygons and the multiple polygons, and the trajectory lines of the n trajectory lines that do not fit polygons are determined as the second mark; If no polygon is fitted through the n trajectory lines, the n trajectory lines are determined as the second marks.

5. The method according to any one of claims 1 to 4, characterized in that: The identifying the target content from the screen content of the electronic device according to the first area circled by the first mark includes: If the content in the first area of ​​the screen content of the electronic device is text content, determining that the text paragraph in which the text content in the first area is located is the first paragraph; Determine the degree of overlap between each sentence in the first paragraph and the first region based on the coordinates of each sentence in the first paragraph and the coordinate range of the first region, and identify a target sentence from the first paragraph based on the degree of overlap between each sentence in the first paragraph and the first region, where the target sentence is the target content; or determine the degree of overlap between each character in the first paragraph and the first region based on the coordinates of each character in the first paragraph and the coordinate range of the first region, and identify a target character from the first paragraph based on the degree of overlap between each character in the first paragraph and the first region, where the target character is the target content.

6. The method according to any one of claims 1 to 4, characterized in that: The identifying the target content from the screen content of the electronic device according to the first area circled by the first mark includes: If the content in the first area of ​​the screen content of the electronic device is table content, determining that the table to which the table content in the first area belongs is a first table; determining, based on the coordinates of each cell in the first table and the coordinate range of the first area, a degree of overlap between each cell in the first table and the first area; The text in the target cell is identified from the first table according to the overlap between each cell in the first table and the first area, and the text in the target cell is the target content.

7. The method according to any one of claims 1 to 4, characterized in that: The identifying the target content from the screen content of the electronic device according to the first area circled by the first mark includes: If the content in the first area of ​​the screen content of the electronic device is picture content, determining that the picture to which the picture content in the first area belongs is a first picture; The first picture is identified according to the coordinates of the first picture, where the first picture is the target content.

8. The method according to any one of claims 1 to 4, characterized in that: The identifying the target content from the screen content of the electronic device according to the first area circled by the first mark includes: If the content in the first area of ​​the screen content of the electronic device is formula content, the formula in which the formula content in the first area is located is identified, and the formula in which the formula content in the first area is located is the target content.

9. The method according to any one of claims 1 to 4, characterized in that: The identifying target content from the screen content of the electronic device according to the designated mark includes: If the designated mark includes the first mark and the second mark, and the first mark and the second mark do not overlap, retaining the first mark in the designated mark and discarding the second mark, and identifying the target content from the screen content of the electronic device according to the first area circled by the first mark; or If the designated mark includes the first mark and the second mark, and the first mark and the second mark do not overlap, the first mark is discarded in the designated mark and the second mark is retained, and the target content is identified from the screen content of the electronic device based on the second area circled by the second mark or based on the content blocked by the second mark.

10. The method according to any one of claims 1 to 4, characterized in that: The second mark is used to circle the content that does not need to be identified, and identifying the target content from the screen content of the electronic device according to the designated mark includes: If the designated mark includes the first mark and the second mark, and the second mark is located within the first mark, then the other areas in the first area circled by the first mark except the second area circled by the second mark are determined as third areas, and the target content is identified from the screen content of the electronic device based on the third area.

11. The method according to any one of claims 1 to 4, characterized in that: The second mark is used to block content that does not need to be identified, and identifying target content from the screen content of the electronic device according to the specified mark includes: If the designated mark includes the first mark and the second mark, and the second mark is located within the first mark, the target content is identified from the screen content of the electronic device based on the first area circled by the first mark, and the content obscured by the second mark is deleted from the identified target content.

12. The method according to any one of claims 1 to 4, characterized in that: The second mark is used to circle the content that does not need to be identified, and identifying the target content from the screen content of the electronic device according to the designated mark includes: If the designated marker includes the first marker and the second marker, and the first marker intersects the second marker, then retain the first marker and use the intersection of the first marker and the second marker as a new second marker, or retain the second marker and use the union of the second marker and the first marker as a new first marker; The other areas of the first area circled by the first mark except the second area circled by the second mark are determined as third areas, and the target content is identified from the screen content of the electronic device according to the third areas.

13. The method according to any one of claims 1 to 4, characterized in that: The second mark is used to block content that does not need to be identified, and identifying target content from the screen content of the electronic device according to the specified mark includes: If the designated mark includes the first mark and the second mark, and the first mark intersects the second mark, retaining the first mark and taking the portion of the second mark located within the first mark as a new second mark; The target content is identified from the screen content of the electronic device according to the first area circled by the first mark, and the content obscured by the second mark is deleted from the identified target content.

14. The method according to any one of claims 1 to 4, characterized in that: After identifying the target content from the screen content of the electronic device according to the designated mark, the method further includes: Storing the identified target content to the system clipboard; If it is detected that any edit box obtains focus, the target content stored in the system clipboard is pasted into the edit box.

15. A screen content recognition device, characterized in that: The device comprises: A first acquisition module is configured to acquire n trajectory lines drawn on the screen of the electronic device upon receiving a screen content recognition instruction, where n is a positive integer; a second acquisition module, configured to acquire designated marks based on the n trajectory lines, the designated marks including a first mark and / or a second mark, the first mark being used to circle content to be identified, and the second mark being used to circle content not to be identified or to cover content not to be identified; an identification module, configured to identify target content from the screen content of the electronic device according to the designated mark; In which, the recognition module is used to: if the designated mark includes the first mark and does not include the second mark, then identify the target content from the screen content of the electronic device according to the first area circled by the first mark; if the designated mark includes the second mark and does not include the first mark, then identify the target content from the screen content of the electronic device according to the second area circled by the second mark or according to the content blocked by the second mark.

16. A computer device, characterized in that: The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the method according to any one of claims 1 to 14 when executed by the processor.

17. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the method according to any one of claims 1 to 14.

Images