Method for processing content in picture and electronic equipment

Abstract

The invention provides a method for processing content in a picture and electronic equipment. In the method, the electronic equipment can highlight an object in a picture, label an object type corresponding to the object in the picture, and provide a service associated with the object type for the object in the picture. By implementing the scheme provided by the invention, the content contained in the picture can be conveniently obtained, and the content in the picture can be correspondingly processed.

Description

Technical Field

[0001] The present application relates to the field of terminals and image processing, and in particular to a method and electronic device for processing content in a picture. Background Art

[0002] At this stage, users like to record their lives through pictures. Pictures usually include a lot of useful content, but users do not want the content to be in the form of pictures only. They may want to use the content for applications other than pictures. For example, when the content is text, the user may want to copy the text and then paste it into a message box. Therefore, image information processing technology was born to extract meaningful content from pictures, such as text or graphics. Through image information processing technology, electronic devices can better understand and process image content, providing people with smarter and more efficient services.

[0003] How to handle the content in the picture is worth discussing. Summary of the invention

[0004] The present application provides a method and an electronic device for processing the content in an image. For various contents such as objects, texts, documents, cards, QR codes, etc. contained in the image, the method can simultaneously support: detecting and identifying texts in the image, classifying and clipping objects in the image, detecting and parsing QR codes contained in the image, and detecting and separating documents and cards contained in the image.

[0005] In a first aspect, the present application provides a method for processing content in a picture, the method comprising: an electronic device displays a picture; in response to an operation on the picture, the electronic device highlights a first text in the picture and marks the entity type of the first text at the first text; the first entity type is determined by comparing the entity type to be verified corresponding to the first text with at least one entity type associated with the text in a document card to which the first text belongs; the entity type to be verified is determined based on the structure of the first text; in response to the operation on the first text, the electronic device provides service options associated with the entity type of the first text.

[0006] In the above embodiment, the first text can be understood as the structured text in the following embodiment. The electronic device annotating the first text includes the electronic device displaying a label of the entity type of the first text on the first text to prompt the user of the entity type of the first text. At the same time, the entity type of the first text is determined after verifying the entity type corresponding to the text in the document card to which it belongs, which is more accurate. No wrong entity type label will be displayed due to an error in structured text extraction. When the user operates the structured text in the picture, no invalid operation or erroneous operation will occur, thereby ensuring the user's experience.

[0007] In combination with the first aspect, in some embodiments, after the electronic device highlights the first text located on the document card in the picture in response to an operation on the picture, the method further includes: the electronic device identifies the document card from the picture and identifies the first text from the picture; determines the type attribute of the document card and the entity type to be verified of the first text; when it is determined that the first text is located on the document card, the electronic device compares at least one entity type associated with the text in the document card with the entity type to be verified; and the electronic device determines that at least one entity type associated with the text in the document card includes the entity type to be verified.

[0008] In the above embodiments, the electronic device identifies the document card and the text simultaneously. When it is determined that there is a document card, the entity type of the text belonging to the document card is verified by using the entity type of the text in the document card associated with the type attribute of the document card. This improves the accuracy of identifying the entity type of the first text.

[0009] In combination with the first aspect, in some embodiments, when the document card is a table and the document card further includes a second text, the method further includes: the electronic device highlights the second text; when it is determined that the second text and the first text are not in the same row or the same column in the document card, or the entity types of the first text and the second text are different, when highlighting the second text, the entity type corresponding to the second text is also marked at the second text.

[0010] In combination with the first aspect, in some embodiments, when it is determined that the second text and the first text are in the same row or the same column in the document card and the entity types of the first text and the second text are the same, the entity type corresponding to the second text is not marked at the second text in the picture.

[0011] In the above embodiments, implementing the above embodiments can enable the electronic device to label the structured text in the picture with fewer labels when the distribution of the structured text is relatively regular, making the picture more concise, clearer for the user to observe the picture, and improving the user experience.

[0012] In combination with the first aspect, in some embodiments, in response to an operation on the first text, the electronic device provides service options associated with the entity type of the first text, specifically including: in response to a first operation on the first text, the electronic device displays a first prompt on the first side of the screen, and the first prompt is used to prompt the user that there is a service bar hidden on the first side; in response to an operation of dragging the first text to a first hot zone, the electronic device closes the first prompt and displays the hidden service bar, and at least one service option associated with the entity type of the first text is displayed in the service bar, and the distance from the position farthest from the first side in the first hot zone to the first side is less than or equal to a first distance threshold.

[0013] In the above embodiments, an example of the first hot zone can be hot zone 1 in the following specification, an example of the first prompt can be prompt 511 in the following specification. An example of the service bar can be service bar 521 in the following specification.

[0014] Implementing the above embodiments can enable the electronic device to highlight the first text in the image and annotate the content in the image, so that the user can more conveniently observe the content contained in the picture and facilitate the user to select the object to be operated. And when the user drags an object in the picture, a prompt can be used to remind the user that the dragged object can be further operated. It makes the user experience better when processing the content in the picture.

[0015] In combination with the first aspect, in some embodiments, when the first distance threshold is 0, the first hot zone is the first side.

[0016] In the above embodiments, the scenario when the first distance threshold is 0 can refer to the description in the following specification for Figure 5C It can be regarded that the first hot zone includes the area including prompt 511 or is prompt 511 itself.

[0017] Implementing the above embodiments can enable the electronic device to respond to the operation of dragging the first text to the first side and display the service bar on the first side. It enables the user to freely drag objects in the picture within the screen range, and when wanting to perform further operations, it can also be achieved by dragging the target object to the edge of the screen. It improves the user experience.

[0018] In combination with the first aspect, in some embodiments, when the first side is the side edge of the screen of the electronic device and the electronic device displays the hidden service bar, the method further includes: the electronic device reduces the first text to a height of the first text less than the height of the service option.

[0019] In the above embodiments, the scenario where the first side is the side edge of the screen of the electronic device can refer to the description in the following specification forFigure 5C Description.

[0020] Implementing the above embodiments can cause the electronic device to shrink the dragged object (the first text) when displaying the service bar on the side of the screen, enabling the user to avoid the situation where one object covers two service icons simultaneously when dragging an object in the picture to a service icon in the service bar and then releasing the object, thus avoiding possible errors when the user selects a service.

[0021] In combination with the first aspect, in some embodiments, when the first side is the top or bottom edge of the screen of the electronic device and the electronic device displays the hidden service bar, the method further includes: the electronic device shrinks the first text until the length of the first text is less than the length of the service option.

[0022] In combination with the first aspect, in some embodiments, the method further includes: when dragging the shrunk first text to the first service option in the service bar, in response to the operation of releasing the shrunk first text, the electronic device activates the first service.

[0023] In the above embodiments, it can cause the electronic device to shrink the dragged object when displaying the service bar at the top or bottom edge of the screen. When the user drags an object in the picture to a service icon in the service bar and then releases the object, the situation where one object covers two service icons simultaneously will not occur, avoiding possible errors when the user selects a service. This ensures the user experience.

[0024] In combination with the first aspect, in some embodiments, in response to an operation on the picture, when the electronic device highlights the first text in the picture that is located on the document card and marks the entity type of the first text at the first text, the method further includes: the electronic device also highlights the first object in the picture other than the first text, and marks the object type of the first object at the first object; wherein, the first object includes a third text or a graphic main body not in the document card.

[0025] In the above embodiments, the electronic device can provide services for the first text. The user can autonomously select the service they want to use by dragging the first text to the service option in the service bar and then releasing it. This improves the user experience.

[0026] In combination with the first aspect, in some embodiments, when the first object includes a graphic main body, before the electronic device highlights the graphic main body, the method further includes: the electronic device determines that the blur degree of the graphic main body is less than a first preset value, the Euclidean distance between the graphic main body and the center of the picture is less than a second preset value, and the proportion of the graphic main body in the picture is greater than a third preset value.

[0027] In the above embodiments, the electronic device can identify the prominent subject in the picture. Highlighting the prominent subject allows the user to focus their attention on more prominent objects, making the process of obtaining the content in the picture more efficient for the user.

[0028] In combination with the first aspect, in some embodiments, when the first object is a graphic subject, the object types of the first object include people, vehicles, buildings, animals, plants, document cards, and two-dimensional codes; when the first object is text, the object types of the first object include mobile phone numbers, landline numbers, addresses, website addresses, bank card numbers, and express waybills.

[0029] In the above embodiments, the electronic device can identify the object type of the first object. By encompassing common graphic subjects and text types in daily life, the usage scenarios in daily life of the user are optimized, and the user experience is improved.

[0030] In a second aspect, the present application provides a method for processing the content in a picture. The method includes: the electronic device displays a picture; in response to an operation on the picture, the electronic device highlights a first object in the picture and labels the object type of the first object at the first object; in response to a first operation on the first object, the electronic device displays a first prompt on a first side of the screen, and the first prompt is used to prompt the user that a service bar is hidden on the first side; in response to an operation of dragging the first object to a first hot zone, the electronic device displays the hidden service bar, and at least one service option associated with the object type of the first object is displayed in the service bar, and the distance from the position farthest from the first side in the first hot zone to the first side is less than or equal to a first distance threshold.

[0031] In the above embodiments, the first object is a general term for structured text and graphic subjects in the following embodiments. An example of the first hot zone can be hot zone 1 in the following specification, and an example of the first prompt can be prompt 511 in the following specification. An example of the service bar can be service bar 521 in the following specification.

[0032] Implementing the above embodiments, the electronic device highlights the content in the image and labels the content in the image, enabling the user to more conveniently observe the content included in the picture and facilitating the user to select the object for operation. And when the user drags an object in the picture, a prompt can be used to remind the user that further operations can be performed on the dragged object. This makes the user experience better during the process of processing the content in the picture.

[0033] In combination with the second aspect, in some embodiments, when the first distance threshold is 0, the first hot zone is the first side.

[0034] In the above embodiments, the scenario when the first distance threshold is 0 can be referred to in the following specification forFigure 5A The description. The first hot zone can be regarded as the area including the hint 511 or just the hint 511 itself.

[0035] Implementing the above embodiments enables the electronic device to respond to the operation of dragging the first object to the first side and display the service bar on the first side. It allows the user to freely drag the object in the picture within the screen range, and when further operations are desired, it can also be achieved by dragging the target object to the edge of the screen. This improves the user experience.

[0036] In combination with the second aspect, in some embodiments, when the first side is the side edge of the screen of the electronic device and the electronic device displays the hidden service bar, the method further includes: the electronic device shrinks the first object until the height of the first object is less than the height of the service option.

[0037] Implementing the above embodiments enables the electronic device to shrink the dragged object when displaying the service bar on the side edge of the screen, so that when the user drags the object in the picture to the service icon in the service bar and then releases the object, there will be no situation where one object covers two service icons at the same time, avoiding possible errors when the user selects a service.

[0038] In combination with the second aspect, in some embodiments, when the first side is the top edge or the bottom edge of the screen of the electronic device and the electronic device displays the hidden service bar, the method further includes: the electronic device shrinks the first object until the length of the first object is less than the length of the service option.

[0039] Implementing the above embodiments enables the electronic device to shrink the dragged object when displaying the service bar on the top edge or the bottom edge of the screen, so that when the user drags the object in the picture to a service icon in the service bar and then releases the object, there will be no situation where one object covers two service icons at the same time, avoiding possible errors when the user selects a service. This guarantees the user experience.

[0040] In combination with the second aspect, in some embodiments, when the shrunk first object is dragged to the first service option in the service bar, in response to the operation of releasing the shrunk first object, the electronic device activates the first service.

[0041] In the above embodiments, an example of the first service option can be the service option 521a in the following specification.

[0042] Implementing the above embodiments enables the electronic device to provide services for the first object. The user can autonomously select the desired service by dragging the object to the service icon in the service bar and then releasing it.

[0043] In combination with the second aspect, in some embodiments, annotating the object type of the first object at the first object specifically includes: displaying a label corresponding to the object type of the first object on the first object.

[0044] Implementing the above embodiments can enable the electronic device to annotate the object type of the object in the picture. When the user observes the content in the picture, the content in the picture can be classified and integrated more quickly.

[0045] In combination with the second aspect, in some embodiments, the first object includes at least one of a graphic main body or structured text.

[0046] An example of the graphic main body in the above embodiment can be the main body 221 in the following specification, and an example of the structured text can be the text 212 in the following specification.

[0047] Implementing the above embodiments can enable the electronic device to process the graphic main body and structured text in the picture. The user can directly operate on the graphic main body and structured text in the picture. It makes the process of the user obtaining the content in the picture more convenient and improves the user experience.

[0048] In combination with the second aspect, in some embodiments, when the first object includes a first structured text and a second structured text, displaying a label corresponding to the object type of the first object on the first object specifically includes: when the first structured text and the second structured text are in the same row or the same column in the table, and the object types corresponding to the first structured text and the second structured text are the same, the electronic device only displays the label corresponding to the object type on one of the first structured text or the second structured text.

[0049] In the above embodiment, an example of the first structured text can be the text 301 in the following specification, and an example of the second structured text can be the text 302 in the following specification.

[0050] Implementing the above embodiments can enable the electronic device to label the structured text in the picture with fewer labels when the structured text is distributed regularly, making the picture more concise, making it clearer and more understandable for the user to observe the picture, and improving the user experience.

[0051] In combination with the second aspect, in some embodiments, before determining the object type corresponding to the structured text, the method further includes: the electronic device determines the object type to be verified corresponding to the structured text; when it is determined that the structured text is located in a document card, the electronic device compares the object type associated with the text in the document card with the object type to be verified; the electronic device determines that the object type associated with the text in the document card includes the object type to be verified.

[0052] Implementing the above embodiments can enable the electronic device to verify the entity recognition result of the text, avoiding displaying incorrect entity type labels due to incorrect structured text extraction. When the user operates on the structured text in the picture, invalid operations or incorrect operations will not occur, ensuring the user experience.

[0053] In combination with the second aspect, in some embodiments, when the first object includes a graphic main body, before the electronic device highlights the first object, the method further includes: the electronic device determines that the blur degree of the graphic main body is less than a first preset value, the Euclidean distance between the graphic main body and the center of the picture is less than a second preset value, and the proportion of the graphic main body in the picture is greater than a third preset value.

[0054] Implementing the above embodiments can enable the electronic device to confirm the prominent main body in the picture. Highlighting the prominent main body can enable the user to focus on more prominent objects, making the process of the user obtaining the content in the picture more efficient.

[0055] In combination with the second aspect, in some embodiments, when the first object is a graphic main body, the object type of the first object includes people, vehicles, buildings, animals, plants, document cards, and two-dimensional codes; when the first object is structured text, the object type of the first object includes mobile phone numbers, landline numbers, addresses, website addresses, bank card numbers, and express waybills.

[0056] In the above embodiments, the electronic device can confirm the object type of the first object. Incorporating common graphic main bodies and text types in life optimizes the usage scenarios of users in daily life and improves the user experience.

[0057] In a third aspect, an embodiment of the present application provides an electronic device, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, and the memory is used to store computer program code, and the computer program code includes computer instructions. The one or more processors call the computer instructions to enable the electronic device to execute the method implemented in the first aspect.

[0058] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, including instructions, when the instructions run on an electronic device, enabling the electronic device to execute the method implemented in the first aspect.

[0059] In a fifth aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device. The chip system includes one or more processors, and the processors are used to call computer instructions to enable the electronic device to execute the method implemented in the first aspect.

[0060] In a sixth aspect, an embodiment of the present application provides a computer program product including instructions. When the computer program product runs on an electronic device, the electronic device is caused to execute the method implemented in the first aspect.

[0061] It can be understood that the electronic device provided in the third aspect, the computer storage medium provided in the fourth aspect, the chip system provided in the fifth aspect, and the computer program product provided in the sixth aspect are all used to execute the method provided in the embodiments of the present application. Therefore, other beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method, and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 An exemplary user interface showing an electronic device displaying a picture with structured text is shown;

[0063] Figure 2 An exemplary user interface involved when an electronic device processes the content in a picture is shown;

[0064] Figure 3 An exemplary scenario showing an electronic device adding tags to the structured text in a picture is shown;

[0065] Figure 4 An exemplary scenario for an exemplary description of an electronic device providing a service based on the extracted structured text is shown;

[0066] Figure 5A - Figure 5C An exemplary scenario involved when an electronic device provides a service corresponding to a subject is shown by an exemplary description;

[0067] Figure 6 A schematic software structure diagram involved when an electronic device processes the content in a picture is shown;

[0068] Figure 7 An exemplary module interaction diagram when an electronic device processes the content in a picture is shown exemplarily;

[0069] Figure 8 A schematic interaction flowchart between modules when an electronic device processes the content in a picture is shown;

[0070] Figure 9 It is a schematic structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0071] In one solution, the electronic device can extract the structured text in the picture.

[0072] Structured text refers to text with a certain format. This format includes one or more of the following: the format of mobile phone numbers, landline numbers, addresses, names, website addresses, bank card numbers, express waybill numbers, etc. For example, in some regions, the format of mobile phone numbers usually consists of 11 digits (excluding the country code) and starts with 1. As Figure 1 shown, Figure 1 in the picture in the user interface 10 shown, "Zhang XX" in the picture has a name format, "1XXXXX" has a phone number format, and "XXX City, XXX District" has an address format, all of which are structured text.

[0073] Furthermore, after the electronic device extracts the structured text, it will also determine the entity type corresponding to the structured text based on the structure of the structured text. This entity type is the thing referred to by the structured text, and it can also be understood that the entity type can be used to represent the meaning of the structured text. For example, Figure 1 in, the text "1XXXXX" has a phone number format, so the entity type of this text is a phone number.

[0074] It should be understood here that when the electronic device extracts structured text from a picture, it first needs to recognize the text in the picture, and then analyze the recognized text to determine the structured text in the recognized text based on the analysis results. Therefore, when the electronic device extracts structured text from a picture, it will be affected by factors such as the clarity of the picture, resulting in inaccurate recognition of the text, which in turn affects the judgment of the entity type of the structured text.

[0075] Moreover, the above solution can extract the structured text (text content) in the picture, but it cannot extract other content in the picture except for the text.

[0076] To solve the problem in the foregoing solution that when extracting structured text from a picture, it will be affected by factors such as the clarity of the picture, resulting in inaccurate recognition of the text, which in turn affects the judgment of the entity type of the structured text, another method for processing the content in the picture is provided. In this method, the entity type determined by the electronic device based on the structure of the structured text is the entity type to be verified rather than the final entity type. Furthermore, the electronic device will determine whether the structured text belongs to a document card. For the structured text in the document card, the electronic device can verify the entity to be verified of the structured text. After the verification passes, the electronic device will highlight the structured text in the picture and mark the entity type (after verification) of the structured text at the location of the structured text. If the verification fails, the electronic device can highlight the structured text in the picture, but will not mark the entity type of the structured text at the location of the structured text.

[0077] For a case where the structured text does not belong to a document card, the electronic device may use the entity type to be verified of the structured text as the entity type of the structured text. It may also determine that the structured text has no entity type.

[0078] The entity type of the structured text is determined by comparing the entity type to be verified corresponding to the structured text with at least one entity type associated with the text in the document card to which the structured text belongs. Before highlighting the structured text, the entity type of the structured text will be determined. The process includes: The electronic device identifies the document card from the picture and identifies the structured text from the picture. Then, it determines the type attribute of the document card and the entity type to be verified of the structured text. When it is determined that the structured text is located in the document card, the electronic device compares at least one entity type associated with the text in the document card with the entity type to be verified of the structured text. When the electronic device determines that at least one entity type associated with the text in the document card includes the entity type to be verified of the structured text, the electronic device determines that the entity type to be verified of the structured text is the entity type of the structured text.

[0079] In some possible cases, the electronic device may also provide a service corresponding to the entity type based on the entity type corresponding to the structured text.

[0080] Annotating the entity type corresponding to the text in the picture means: Displaying a label corresponding to the entity type of the text (or called an entity type label) on the text to represent the entity type corresponding to the text.

[0081] Using the method for processing the content in the picture, in addition to extracting the structured text, the electronic device can also extract the graphic main body (simply called the main body) in the picture and annotate the main body type of the main body in the picture.

[0082] In some possible cases, the electronic device may also provide a service associated with the main body type of the main body based on the main body type of the main body.

[0083] Annotating the main body type of the main body in the picture means: Displaying a label corresponding to the main body type of the main body (or called a type label) on the text to represent the main body type corresponding to the main body.

[0084] The main body type of the main body can be one of a person, an animal, a plant, a vehicle, a building, a document card, a two-dimensional code, etc. A main body is a connected graphic in the picture. It can also be understood that there is no connectivity between two main bodies.

[0085] Among them, having connectivity means that there is at least one path connecting each pair of nodes in a graphic.

[0086] The subject may be a prominent subject, or may include a prominent subject and a non-prominent subject.

[0087] Among them, the prominent subject is the subject in the body whose significance is higher than or equal to the preset significance threshold. The non-prominent subject is the subject in the body whose significance is lower than the preset significance threshold. The significance of the subject is related to one or more of factors such as the position of the subject in the picture, the area of the subject in the picture, and the degree of blurriness of the subject.

[0088] The Euclidean distance between the subject and the center of the picture is negatively correlated with the significance of the subject; the area of the subject in the picture is positively correlated with the significance of the subject; the degree of blurriness of the subject is negatively correlated with the significance of the subject.

[0089] In some possible cases, when the Euclidean distance between a subject in the picture and the center of the picture is smaller, the area of the subject in the picture is larger, and the degree of blurriness is smaller, the greater the possibility that the electronic device determines the subject as a prominent subject, and the smaller the possibility of determining the subject as a non-prominent subject, but the greater the possibility of determining the subject as a prominent subject.

[0090] At this time, the process of determining a subject as a prominent subject includes: determining that the degree of blurriness of the graphic subject is less than a preset value 1, the Euclidean distance between the graphic subject and the center of the picture is less than a preset value 2, and the proportion of the graphic subject in the picture is greater than a preset value 3. The preset value 1, preset value 2, and preset value 3 are values preset in the electronic device. In some possible cases, the preset value 1, preset value 2, and preset value 3 can be changed by the user.

[0091] Among them, the method for judging the degree of blurriness of the subject includes but is not limited to: judging by the local contrast of the picture part where the graphic subject is located. If the local contrast is higher than a preset value 4, it can be judged that the graphic subject is clear, that is, the degree of blurriness of the graphic subject is less than a preset value 1.

[0092] In the following, taking the subject as a prominent subject as an example for illustration. Without special instructions, the subject mentioned in the following is the prominent subject. The relevant content when the subject includes a prominent subject and a non-prominent subject can refer to the following description and will not be elaborated here.

[0093] Next, the exemplary scenarios involved when the electronic device extracts the content in the picture from the picture and the exemplary scenarios for pulling up services based on the extracted picture content are introduced.

[0094] First, based on Figure 2 Exemplary scenarios involved when the electronic device extracts the content in the picture from the picture are described exemplarily.

[0095] Such as Figure 2As shown in (1), the user interface 20 is an exemplary user interface when the gallery application of the electronic device displays a picture. In response to an operation on the picture (such as a two-finger press operation), the electronic device can extract the prominent subject and structured text in the picture, and label the subject type of the prominent subject and the entity type corresponding to some or all of the structured text.

[0096] During the process of the electronic device processing the content in the picture, an icon 202 as shown in (1) can be displayed. This icon 202 indicates that the electronic device is processing the content in the picture. For the display of the extraction result and the annotation result, reference can be made to the user interface 21 shown in (2) below. Figure 2 Figure 2 As shown in (2), the user interface 21 is an exemplary user interface displayed after the extraction and annotation of the image content in the picture. The extracted image content in this picture can include: text 212, text 213, subject 221, subject 222, subject 223, and subject 224, etc. Among them, text 212 has a format corresponding to a phone number, and text 213 has a format corresponding to an address, both belonging to structured text. Subjects 221 - 224 are relatively clear and centered in the picture and can be regarded as prominent subjects.

[0097] For the image content extracted from the picture, the electronic device can highlight this image content in the picture. For example, referring to (2), the text 212, text 213, subject 221, subject 222, subject 223, and subject 224 are made bold to achieve highlighting. The electronic device can add labels to the prominent subjects in the picture to label the corresponding subject types of the prominent subjects. Referring to (2), the subject type corresponding to subject 221 is an animal, and the electronic device can add an animal label 221a to subject 221. The subject type corresponding to subject 222 is a person, and the electronic device can add a person label 222a to subject 222. The subject type corresponding to subject 223 is a vehicle, and the electronic device can add a vehicle label 223a to subject 223. The subject type corresponding to subject 224 is a QR code, and the electronic device can add a QR code label 224a to subject 224. Figure 2

[0098]

[0098] Figure 2 Figure 2 Figure 2 Figure 2

[0099] The electronic device can also add labels to the structured text in the picture to label the entity type corresponding to the text. Referring to (2) Figure 2In (2), if the entity type corresponding to the text 212 is a telephone number, the electronic device may add a telephone label 212a to the text 212. If the entity type corresponding to the text 213 is an address, the electronic device may add an address label 213a to the text 213.

[0100] It should be noted here that the rules for displaying labels of entity types in pictures include but are not limited to one of the following display rules:

[0101] Display Rule 1. Reference can be made to Figure 3 In (1), the electronic device displays a label of the entity type corresponding to the text on each text in the structured text.

[0102] Display Rule 2. Reference can be made to Figure 3 In (2), for at least two texts corresponding to the same entity type, and when the at least two texts are in the same row or the same column in a table, the electronic device only selects to display the label of the entity type corresponding to the text on one of the at least two texts. For example, the entity types corresponding to the text 301, the text 302, and the text 303 are all addresses, and the text 301 - text 303 are in the same column in the table. At this time, the electronic device only selects to display the label of the entity type corresponding to the text 301 on the text 301.

[0103] Display Rule 3. The electronic device displays a label of the entity type corresponding to the text around each text in the structured text. The surroundings of a text include positions where the distance from the center of the text is less than a preset distance value.

[0104] In addition to the significant subject and the structured text, the image content extracted by the electronic device may also include other content. For example, part or all of the text in the picture except the structured text. For example, Figure 2 The text 211 shown in (2) does not belong to the structured text, but can still be extracted.

[0105] It should be understood here that the foregoing bolding of the extracted text is for illustrative purposes, and other forms such as adding highlights or masks may also be used.

[0106] It should also be understood that the foregoing Figure 2 In (2), the extraction of the significant subject is taken as an example for illustration. In actual situations, the significant subject and the non-significant subject can also be extracted and labeled simultaneously. For example, Figure 2 The picture shown in (1) includes a non-significant subject: the subject 201. In the scenario of simultaneously extracting and labeling the significant subject and the non-significant subject, when an operation on the picture (such as a two-finger press operation) is detected, in response to the operation, what the electronic device displays is notFigure 2 the user interface 21 shown in (2), and can display Figure 2 the user interface 22 shown in (3).

[0107] Reference Figure 2 to the user interface 22 shown in (3), compared with Figure 2 the user interface 21 shown in (2), the main body 201 in the picture is extracted, and a label of the main body type of the main body 201, i.e., the building label 201a, is displayed on the main body 201.

[0108] It should be noted that Figure 2 the example is described with the image content extracted by the electronic device including structured text and the main body. In actual situations, the electronic device can only extract the structured text in the picture, or only extract the main body in the picture. The relevant process can refer to the description of the relevant content in Figure 2 above, and will not be elaborated here.

[0109] In some possible cases, since the graphic of the QR code itself does not have much information, but the scanning of the QR code is more valuable. Therefore, for the label corresponding to the QR code, instead of displaying the main body type of the QR code involved above, other more useful prompts can be further displayed. For example, the icon of the application for scanning the QR code can be displayed on the QR code. For example, referring to Figure 2 the user interface 25 shown in (4), the electronic device recognizes that the main body 224 can be scanned through the application 1, and the icon 224b of the application 1 can be displayed on the main body 224. Or, the icon of the source application of the QR code can also be displayed. The source application of the QR code refers to the application that generates the QR code.

[0110] It should also be noted here that the electronic device extracts the structured text in the picture means that: after the electronic device recognizes the structured text in the picture, it converts the structured text in the picture into an operable object, so that the extracted structured text can receive the operations of the user. In response to the operation on the structured text, the electronic device can provide services corresponding to the text. Extracting the structured text in the picture can also be understood as separating the structured text from the picture.

[0111] Among them, the services corresponding to the text include but are not limited to one or more of the following services: basic services for the text and services corresponding to the entity type of the structured text. Refer to Figure 4In the user interface 40, in response to an operation on the text 213, the electronic device can display the control 2131 and the control 2132. Among them, the control 2132 is used to provide basic services for the text 213: copying the text 213. The control 2131 is the service corresponding to the entity type of the text 213. The entity type corresponding to the text 213 is an address, and it can provide services such as opening a navigation application and performing navigation.

[0112] It should also be noted that when the electronic device extracts the main body in the picture, it means that after identifying the main body, the main body is converted into an operable object so that the extracted main body can receive user operations, and the operations can include drag operations. In response to an operation on the main body, the electronic device can provide services corresponding to the main body. Extracting the main body in the picture can also be understood as separating the main body from the picture.

[0113] Among them, the services corresponding to the main body include, but are not limited to, one or more of the following services: saving, searching, favoriting, sharing, etc. The following combines Figure 5A Describe an exemplary scenario of providing services associated with the entity type corresponding to the extracted main body based on the extracted main body.

[0114] Refer to Figure 5A In the user interface 50 shown in (1) below, in response to a drag operation of dragging the main body 221 at position 1 to position 2, the electronic device can display the main body 221 at position 2 and continue to display the main body 221 at position 1. It can be understood that the electronic device displays a copy 2211 of the main body 221 at position 2 in the picture (the copy 2211 can still be called the main body 221 in the following text).

[0115] It can also be understood that the drag operation on the main body is not used to change the position of the main body in the picture. The main body will always be displayed at position 1 in the picture.

[0116] After the main body 221 is dragged, the electronic device can also display a prompt 511 on the side of the screen to prompt the user that there is a service bar hidden on this side. The service bar includes at least one service option, and any service option can provide services for the main body 221.

[0117] After the main body 221 is dragged to the hot zone 1, the electronic device can display the hidden service bar. The distance from the position farthest from the side in the hot zone 1 to the side is less than or equal to the distance threshold 1.

[0118] When the distance threshold 1 is equal to 0, the hot zone 1 is the side where the prompt 511 is displayed. Or it can be understood that when the distance threshold 1 is equal to 0, the hot zone 1 is the area where the prompt 511 is located. Refer to Figure 5AThe user interface 51 shown in (2). In response to the operation of dragging the main body 221 to the prompt 511, the electronic device closes the prompt 511 and displays a hidden service bar on the side where the prompt 511 is displayed. For an example of the service bar, reference can be made to the description of (3) below Figure 5A in (3).

[0119] As Figure 5A shown in the user interface 52 in (3), the service bar 521 may include at least one service option associated with the main body type corresponding to the main body 221. When the electronic device displays the service bar 521, it also displays the main body 221 on the service bar 521, and reduces the main body 221 so that the height of the main body 221 is less than or equal to the height of the service option in the service bar 521.

[0120] After the electronic device displays the service bar 521, the electronic device may drag the reduced main body 221 to any service option in the service bar 521. In response to the operation of releasing the reduced main body 221, the electronic device may provide a service corresponding to the service option. For example, referring to Figure 5A the user interface 53 shown in (4) in the figure, the electronic device drags the main body 221 to the service option 521a. In response to the operation of releasing the reduced main body 221, it may provide a service corresponding to the service option 521a. The service option 521a is to save a picture. Therefore, the electronic device may provide the service of saving the main body 221 as a new picture in the gallery service.

[0121] It should be understood that the manner involved in providing the service corresponding to the main body shown above is for illustrative purposes, and in actual situations, it may also be other ways. For example, in addition to the drag operation, the electronic device may also display the prompt 511 based on the operation of long pressing the main body 221. After the prompt 511 is displayed, the service bar 521 is displayed based on the drag operation. Further select the service option for the main body 221. For another example, referring to Figure 5A the user interface 54 shown in Figure 5B . In response to the long press operation on the main body 221, the electronic device may not display the prompt 511, but display the service bar 521, and then select the service option for the main body 221 (such as the service option 521a). When selecting the service option for the main body 221, it may be the aforementioned drag operation or directly clicking on the service option. Figure 5B shown in the user interface 54, in response to the long press operation on the main body 221, the electronic device may not display the prompt 511, but display the service bar 521, and then select the service option for the main body 221 (such as the service option 521a). When selecting the service option for the main body 221, it may be the aforementioned drag operation or directly clicking on the service option.

[0122] The aforementioned Figure 5A and Figure 5BThe above description takes as an example the display of a prompt (such as prompt 511) and a service bar (such as service bar 521) on the side of the screen. In actual situations, they may not be displayed on the side, but rather on the bottom or top edge. When the prompt and the service bar are displayed on the bottom or top edge, the description of shrinking the main body is changed to that the length of the shrunk main body is less than or equal to the length of the service icon.

[0123] It should be noted that the above Figure 5A and Figure 5B show the display of prompts and service bars based on operations on the main body. In fact, it is also possible to display prompts and service bars based on structured text. The process is the same as the process described above for displaying prompts and service bars based on operations on the main body, except that the main body is changed to text.

[0124] When providing services based on structured text, the relevant content can refer to the following description of Figure 5C .

[0125] As Figure 5C shown in (1) of , the user interface 55 is an exemplary interface displayed after extracting and annotating the image content in the picture.

[0126] Referring to Figure 5C shown in (2) of , in response to the operation of dragging the text 551 at position 3 to position 4, the electronic device can display the prompt 511 on the side of the screen.

[0127] After the text 551 is dragged to the hot zone 1, the electronic device can close the prompt 511 and display the hidden service bar on the side of the screen. For example, referring to Figure 5C shown in (3) of , when the area where the prompt 511 is located is the hot zone 1, when the text 551 is dragged onto the prompt 511, the service bar 521 can be displayed. Moreover, when the service bar is displayed on the side of the screen, the electronic device can also shrink the text 551 so that the height of the text 551 is less than the height of the service option. The shrunk text 551 can be referred to as the text 551 in the user interface 58 shown in (4) of Figure 5C .

[0128] Referring again to Figure 5C shown in (4) of , after the electronic device displays the service bar 521, the electronic device can drag the shrunk text 551 to any service option in the service bar 521. In response to the operation of releasing the shrunk text 551, the electronic device can provide the service corresponding to the service option.

[0129] Among them, for the prompt 511, the hot zone 1, and the service bar 521, reference can be made to the relevant content description in the above Figure 5A , and details will not be elaborated here.

[0130] Figure 6 It shows an exemplary software structure block diagram involved when an electronic device implements a method for processing content in a picture.

[0131] The layered architecture divides software into several layers, and each layer has a clear division of labor. The layers communicate with each other through software interfaces. In some embodiments, the system is divided into five layers, namely, the application layer, the system framework layer, the system library, the hardware abstraction layer, and the kernel layer from top to bottom.

[0132] Among them, the application layer may include a series of application packages. The application package may include system applications, or may also include third-party applications.

[0133] Among them, the system applications may include an image content processing application (which may also be referred to as an application). The system applications may also include applications such as a calendar, a gallery, a call, a navigation, and a note. The image content processing application may also be referred to as an image content processing application package (android application package, APK).

[0134] The third-party applications may include one or more of applications such as music applications and video applications.

[0135] Among them, the image content processing APK application is used to implement the call of the following image content processing algorithms by applications such as the gallery and the camera, so that the extraction and annotation of image content can be realized in applications such as the gallery and the camera.

[0136] The system framework layer (which may also be referred to as the application framework layer) provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. For example, the system framework layer may include a resource manager, a notification manager, a view system, and an activity manager.

[0137] The system framework layer may also include a drag framework, a service transfer framework, and a camera framework.

[0138] Among them, the drag framework can be called by applications such as the gallery and the camera to implement the drag of the main body or entity type in the picture.

[0139] The service transfer framework can be used to determine the service associated with the dragged content (main body or entity type), and pull up and respond to the service.

[0140] The camera framework is used to provide system support for the viewfinder preview stream and manage the life cycle and data acquisition of the camera application.

[0141] The system library may include multiple functional modules. For example, the c library, the openCV (open source computer vision library) computer vision library in classes, the cameraservice, etc., and modules such as the Android Runtime.

[0142] The system library may also include an image content processing algorithm module, which may include a main body processing module, a text processing module, a position discrimination module, a document and card identification module, and a QR code identification module.

[0143] Among them, the main body processing module can be used to identify the main body in the picture and extract the main body from the picture. The main body processing module may include a main body detection module, a main body classification module, and a main body matting module.

[0144] The main body detection module is used to detect the main body in the picture and transmit it to the following main body classification module.

[0145] The main body classification module is used to identify the detected main body and determine its corresponding main body type.

[0146] The main body matting module is used to extract the main body from the picture.

[0147] The hardware abstraction layer is an interface layer located between the kernel layer (not shown) and the hardware layer, and its purpose is to abstract the hardware and provide a virtual hardware platform for the operating system.

[0148] The text processing module is used to extract the text in the picture and identify the entity type corresponding to the structured text. The text processing module includes a text recognition module, an entity type confirmation module, and a text extraction module.

[0149] The text recognition module is used to recognize the text in the picture and transmit it to the following entity type confirmation module.

[0150] The entity type confirmation module is used to identify the entity type corresponding to the structured text.

[0151] The text extraction module is used to extract the text in the picture.

[0152] The position discrimination module is used to confirm the position relationship between the structured text in the picture and the document and card, and transmit the position relationship confirmation result to the text processing module.

[0153] The document and card identification module is used to identify the main body type of the document and card and confirm the information of the document and card.

[0154] The QR code identification module is used to identify the information of the QR code.

[0155] The hardware abstraction layer may include abstraction layers for different hardware. For example, a camera abstraction layer, a sensor abstraction layer, an audio abstraction layer, etc.

[0156] Among them, the camera abstraction layer can be used to abstract the image sensor (the sensor in the camera), and perform various calculations based on the images collected by the image sensor to obtain output results.

[0157] The kernel layer is the layer between the hardware ( Figure 6 not shown) and the software. The kernel layer at least includes a display driver, a camera driver, an audio driver, and a sensor driver. It may also include a power management module.

[0158] Among them, the camera driver can be used to drive the camera to collect images.

[0159] Next, in combination with Figure 6 and Figure 7 the detailed process of each module of the electronic device extracting and annotating the image content in the picture will be described in detail.

[0160] Reference can be made to Figure 7 , Figure 7 which exemplarily shows an exemplary module interaction diagram when the electronic device processes image content.

[0161] As Figure 7 shown, the modules involved when the electronic device processes the image content in the picture include: an image content processing APK, a main body processing module, a QR code recognition module, a document and certificate recognition module, and a text processing module. Among them, the interaction process of each module can refer to the following description of steps S11 - step S22.

[0162] S11. The image content processing APK receives an instruction for information extraction and obtains a picture in the picture library.

[0163] In some possible cases, the instruction for information extraction can be an instruction sent by the picture library to the image content processing APK after receiving an operation.

[0164] In some other possible cases, the image content processing APK can be called by the picture library, and the operations on the picture library can be received by the image content processing APK.

[0165] Among them, the operations can refer to the operations involved in (1) in the foregoing Figure 2 .

[0166] It should be understood that the picture library is an example of the source from which the image content processing APK obtains pictures. The electronic device can also obtain pictures from other applications. For example, applications such as cameras and notes.

[0167] After the image content processing APK obtains a picture, it can simultaneously call the following main processing module and text processing module to process the main body and text in the picture respectively. Among them, the relevant content involved in the main body processing by the main body processing module can refer to the following descriptions of steps S12a - S14a and S151b - S153b. The relevant content involved in the text processing by the text processing module can refer to the following descriptions of steps S12b - S14b and S153c - S154c.

[0168] S12a. The main body processing module determines whether there is a main body in the picture.

[0169] In some possible cases, the main body processing module can divide the pixels in the picture into different regions. If there is a region whose features are different from those of the surrounding regions, it can be determined that there is a main body in the picture. If there is no region whose features are different from those of the surrounding regions, it can be determined that there is no main body in the picture. Among them, the features of the region include one or more of contrast, texture, color, etc.

[0170] The main body processing module can use image segmentation algorithms to divide the pixels in the picture into different regions. For example: region - based watershed algorithm, edge - based Canny algorithm, semantic segmentation algorithm, etc.

[0171] In addition to the above methods, the determination method can also include other methods: for example, using object detection algorithms such as deep - learning - based object detection algorithms, the position and category of the main body can be detected in the picture through a pre - trained model.

[0172] When it is determined that there is a main body in the picture, the main body processing module executes the following steps S13a - step S153c to process the main body.

[0173] When it is determined that there is no main body in the picture, the main body processing module no longer executes the following steps S13a - step S153c.

[0174] S13a. The main body processing module obtains the main body region from the picture.

[0175] The main body region is a rectangular region where the main body is located. One main body region can include one main body.

[0176] S14a. The main body processing module determines the main body type of the main body in the main body region.

[0177] In some possible cases, at least one preset subject feature and its corresponding subject type are recorded in the subject processing module. When the subject processing module determines that the feature of the subject in the subject area is more similar to a preset subject feature than a similarity threshold, it can determine that the subject is the subject type corresponding to the preset subject feature. Among them, the preset subjects include one or more of people, animals, plants, vehicles, buildings, document cards, two-dimensional codes, etc.

[0178] For example, the preset subject feature can be encapsulated into a subject detection model. The subject detection model is built based on a neural network. The subject processing module can use the subject detection model to extract the feature of the subject in the subject area, perform comparison and other processing based on the feature of the subject in the subject area and the preset subject feature, and output the confidence of the subject in the subject area with each preset subject feature. Subsequently, the subject processing module determines the maximum confidence from the confidences of the subject in the subject area with each preset subject feature. When the maximum confidence is greater than the similarity threshold, it determines the subject type corresponding to the preset subject feature corresponding to the maximum confidence as the subject type corresponding to the subject.

[0179] It should be noted here that after the subject processing module determines the subject type of the subject in the subject area, it can output the subject type corresponding to the subject to the image content processing APK.

[0180] After the subject processing module executes step S14a to obtain the subject type of the subject, the subject processing module can execute step S151a to send the two-dimensional code area to the two-dimensional code recognition module. Subsequently, the two-dimensional code recognition module performs two-dimensional code processing to obtain two-dimensional code information and transmits the two-dimensional code information to the image content processing APK. The subject processing module can also execute step S151b to screen out the area with significant subjects in the subject area. Subsequently, the subject processing module determines the edge information of the significant subject and transmits the edge information of the significant subject to the image content processing APK. The subject processing module can also execute step S151c to send the document card area to the document card recognition module. Subsequently, after the document card recognition module obtains the format information corresponding to the document card, it transmits the format information to the text processing module. The text processing module validates the entity type to be verified corresponding to the text based on the format information and sends the position information and entity type of the structured text that passes the validation to the image content processing APK.

[0181] Subsequently, referring to the following step S21, the image content processing APK can extract the prominent subject from the picture based on the edge information corresponding to the prominent subject, and extract the structured text from the picture based on the position information corresponding to the structured text. Referring to the following step S22, the image content processing APK can also display the label of the subject type corresponding to the prominent subject on the picture based on the subject type corresponding to the subject and the QR code information corresponding to the QR code, and display the label of the entity type corresponding to the structured text on the picture.

[0182] The process of the QR code recognition module obtaining the QR code information is described below based on steps S151a - S153a.

[0183] S151a. The subject processing module outputs the QR code area to the QR code recognition module.

[0184] It should be noted here that the condition for executing S151a is that the subject type of the subject determined in step S14a includes a QR code.

[0185] S152a. The QR code recognition module determines the QR code information of the QR code in the QR code area.

[0186] The QR code information includes one or more of the scannable application of the QR code, the source application of the QR code, the security of the QR code, etc.

[0187] Among them, the scannable application of the QR code refers to the application that can scan the QR code.

[0188] S153a. The QR code recognition module outputs the QR code information corresponding to the QR code to the image content processing APK.

[0189] This QR code information can be used by the image content processing APK to determine the entity type corresponding to the QR code. For the content of this process, reference can be made to the following description of step S21, which will not be elaborated here.

[0190] The process of the subject processing module screening out the area with the prominent subject in the subject area can be referred to the following description of steps S151b - S153b.

[0191] S151b. The subject processing module screens out the area with the prominent subject in the subject area.

[0192] The subject processing module determines the saliency of the subject in the subject area, and determines the area where the saliency of the subject is higher than or equal to the preset saliency threshold as the area with the prominent subject.

[0193] For the process of determining that the saliency of the subject is higher than or equal to the preset saliency threshold, reference can be made to the previous description of this part of the content, which will not be elaborated here.

[0194] It should be noted that in addition to the methods for determining the saliency of the subject described above, the subject processing module can determine the saliency of the subject through other methods. For example, the saliency of the subject is obtained based on a saliency detection algorithm. Among them, the saliency detection algorithm can be algorithms such as the Itti algorithm, Ft algorithm, Hc algorithm, or Gr algorithm.

[0195] S152b. Determine the edge information of the salient subject.

[0196] The edge information of the subject includes the positions of the pixels at the edge of the subject in the picture.

[0197] The subject processing module uses a matting algorithm to determine the edge information of the salient subject. The matting algorithm can be algorithms such as graph cut-based algorithms, deep learning-based algorithms, or attention mechanisms.

[0198] S153b. The subject processing module outputs the edge information corresponding to the salient subject to the image content processing APK.

[0199] The process by which the document card identification module obtains the format information corresponding to the document card can refer to the following description of steps S151c - S154c.

[0200] The process by which the document card identification module and the text processing module verify the entity types of the text can refer to the following description of steps S151c - S154c.

[0201] S151c. The subject processing module outputs the document card area to the document card identification module.

[0202] The document card area includes the document card. The document card is a general term for documents, cards, and certificates. Among them, the card can include bank cards, credit cards, etc. The certificate can include ID cards, student cards, driver's licenses, passports, and business licenses, etc. The document includes electronic documents and paper documents.

[0203] It should be noted here that the condition for executing S151a is that the subject type of the subject determined in step S14a includes a document card.

[0204] S152c. The document card identification module obtains the format information corresponding to the document card.

[0205] The electronic device first determines the type attribute of the document card, and this type attribute indicates the function and use of the document card, and is used to explain whether the document card is an ID card, a bank card, or a card of other type attributes.

[0206] The format information of the document card includes the entity type corresponding to the text in the document card and the position of the text in the picture. Regarding the entity type, reference can be made to the relevant descriptions above, and details will not be elaborated here.

[0207] The entity types in the text area of the document card are preset in the document card recognition module and are associated with the type attributes of the document card. For example, when the document card is an ID card, the preset entity types in this document card include ID number, name, address, etc.

[0208] The process of obtaining the entity types in the text area of the document card includes: The document card recognition module first determines the attributes of the document card in this area based on the document card area. The attributes of the document card are used to indicate what kind of document the document card is, such as an ID card or a bank card. Further, based on the attributes corresponding to the document card, the entity types of the text area associated with the attributes are obtained.

[0209] It should be noted here that the process for the electronic device to determine the type of the document card includes but is not limited to: extracting the features of the document card (such as color features, texture features, etc.), and inputting the extracted features of the document card into the trained attribute classification model to determine the type attributes of the document card. Among them, the training process of the attribute classification model includes: extracting the features of the document card samples for training, using the attribute classification model to be trained (such as a neural network model) to classify the document card samples based on the extracted features of the document card samples, and determining the predicted type attributes of the document card samples. Then, based on the difference between the predicted type attributes and the marked type attributes corresponding to the document card samples, the parameters in the attribute classification model to be trained are corrected. Training the attribute classification model to be trained with a large number of document card samples can obtain the trained attribute classification model.

[0210] S153c. The text processing module verifies the entity types to be verified corresponding to the structured text based on the format information to obtain the entity types corresponding to the structured text.

[0211] The text processing module determines the document card to which the structured text belongs. It compares the entity types to be verified corresponding to the structured text with the format information of the document card. If the format information of the document card includes the entity types to be verified corresponding to the structured text, it is determined that the entity types to be verified corresponding to the structured text are the entity types corresponding to the structured text. If the format information of the document card does not include the entity types to be verified corresponding to the structured text, it is determined that the entity types to be verified corresponding to the structured text are not the entity types corresponding to the structured text, that is, the structured text has no corresponding entity types and is a structured text with an incorrect entity type recognition.

[0212] When it is determined that the structured text determined by the text processing module does not belong to a document certificate, the entity type to be verified corresponding to the structured text is not compared with the format information of the document certificate. The text processing module directly uses the entity type to be verified corresponding to the structured text as the entity type of the structured text. Alternatively, the entity type to be verified can be ignored, that is, it is confirmed that the structured text has no corresponding entity type.

[0213] Among them, the process of determining the document certificate to which the structured text belongs includes: judging whether the structured text completely intersects with the document certificate based on the smallest rectangular area where the structured text is located and the smallest rectangular area where the document certificate is located.

[0214] The process of determining that the structured text does not belong to the document certificate includes: judging whether the structured text partially intersects or does not intersect with the document certificate based on the smallest rectangular area where the structured text is located and the smallest rectangular area where the document certificate is located.

[0215] It should be noted here that in the case where the entity type to be verified corresponding to the structured text is misrecognized, this step S153b can discriminate the error. Among them, the cases of misrecognizing the entity type to be verified include, but are not limited to, reasons such as the inaccuracy of the recognition algorithm itself or the unclear text in the document certificate.

[0216] Next, based on Figure 8 An exemplary description of the process of verifying the entity type to be verified of the structured text when the structured text in the picture is unclear.

[0217] Refer to Figure 8 , the picture includes a bank card 311. The structured text extracted from the picture includes text 311a: "XX Bank", and text 311b: "52b8 xxxx xx2". The text processing module determines that the entity type to be verified corresponding to the text 311a is a bank card number, and determines that the entity type to be verified corresponding to the text 311b is an express waybill number. The text processing module obtains the format information included in the bank card 311 as: two formats of bank name and bank card number.

[0218] Subsequently, the text processing module determines that text 311a and text 311b belong to the bank card 311. Then, the text processing module performs verification based on the entity type to be verified corresponding to text 311a and the entity type to be verified corresponding to text 311b (express delivery order number) against the two formats of bank name and bank card number. The verification results include: the entity type to be verified corresponding to text 311a (bank name) matches the format information in the bank card, that is, the format information of bank card 311 includes the bank name. The entity type to be verified corresponding to text 311b (express delivery order number) does not match the format information in the bank card, that is, the format information of bank card 311 does not include the express delivery order number. Then, the text processing module determines that the entity type to be verified of text 311a is the entity type of this text 311a. It is determined that the entity type to be verified of text 311b is incorrect, and the entity type corresponding to this text 311b is not output.

[0219] Then, the entity type corresponding to the structured text output by the text processing module is the bank name corresponding to text 311a: "XX Bank".

[0220] It should be noted here that the entity type to be verified corresponding to the text involved in step S153c is determined through the following steps S12b, S13b, and S14b.

[0221] S12b. The text processing module determines whether there is text in the picture.

[0222] If the text processing module determines that there is text in the picture, then the following steps S13b and S14b are executed.

[0223] If the text processing module determines that there is no text in the picture, then steps S13b and S14b are no longer executed.

[0224] S13b. The text processing module identifies the structured text in the picture.

[0225] Identifying the structured text in the picture includes: determining the structured text in the picture and the position of the structured text in the picture.

[0226] S14b. The text processing module determines the entity type to be verified corresponding to the structured text.

[0227] In some possible cases, at least one preset text format and its corresponding entity type are recorded in the text processing module. When the text processing module determines that the similarity between the format of the structured text and a preset text format is greater than the similarity threshold, it can be determined that the entity type of the structured text is the entity type corresponding to the preset text format. Among them, the preset text formats include, but are not limited to, one or more of mobile phone numbers, landline numbers, addresses, names, website addresses, bank card numbers, express delivery order numbers, etc.

[0228] For example, the preset text format can be encapsulated into an entity type detection model. The entity type detection model is built based on a neural network. The entity type processing module can use the entity type detection model to extract the format features of the structured text, and based on the comparison and other processing of the format features of the structured text with the preset text format, output the similarity confidence levels between the structured text and each preset text format. Subsequently, the text processing module determines the maximum confidence level from the confidences of the formats of the structured text and each preset text format. When the maximum confidence level is greater than the similarity threshold, the entity type corresponding to the preset text format corresponding to the maximum confidence level is determined as the entity type corresponding to the structured text.

[0229] It should be understood here that after the text processing module finishes executing step S14b, it waits for the document card verification recognition module to send the format information corresponding to the document card for executing step S153b. When the waiting time exceeds the preset time, the text processing module outputs the entity type corresponding to the text confirmed in step S14b to the image content processing APK, and does not execute step S153b.

[0230] S154c. The text processing module outputs the position information and entity type corresponding to the structured text that has passed the verification to the image content processing APK.

[0231] After the image content processing APK obtains the data sent by the foregoing modules (including data such as the entity type of the main body), it can execute the following step S21 and step S22 based on this data. Extract the content from the picture and annotate the content.

[0232] S21. The image content processing APK extracts the significant main body from the picture based on the edge information corresponding to the significant main body, and extracts the structured text from the picture based on the position information corresponding to the structured text.

[0233] In step S21, the results of extracting the significant main body from the picture and extracting the structured text from the picture can be referred to the relevant description of (2) above, which will not be elaborated here. Figure 2 Hereinabove.

[0234] S22. The image content processing APK displays the label of the entity type corresponding to the significant main body on the picture based on the entity type corresponding to the main body and the two-dimensional code information corresponding to the two-dimensional code, and displays the label of the entity type corresponding to the structured text on the picture.

[0235] The image content processing APK module records tags corresponding to the subject type of the subject and tags corresponding to the entity type of the structured text. The image content processing APK module marks the subject type of the prominent subject on the prominent subject by displaying tags based on the subject type corresponding to the prominent subject. It also marks the entity type corresponding to the structured text by displaying tags based on the entity type corresponding to the structured text.

[0236] Among them, the forms of the tags include, but are not limited to, one or more of text, icons, and animations. When the form of the tag is in the form of an icon, the relevant description of Figure 2 can be referred to, and details will not be repeated here.

[0237] When the QR code information includes the scannable application of the QR code, the electronic device can also add the icon corresponding to the application 1 that can scan the QR code to the QR code body based on the application 1 included in the QR code information corresponding to the QR code. Here, the description of Figure 2 in (4) can be referred to.

[0238] When the QR code information includes the source application of the QR code, the electronic device can also add the icon of the source application to the QR code.

[0239] First, the exemplary electronic device provided by the embodiments of the present application will be introduced below.

[0240] Figure 9 is a schematic structural diagram of the electronic device provided by the embodiments of the present application.

[0241] Below, the embodiments will be specifically described by taking the electronic device as an example. It should be understood that the electronic device may have more or fewer components than Figure 9 shown, may combine two or more components, or may have different component configurations. Figure 9 The various components shown in

[0242] An electronic device may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0243] It can be understood that the structure illustrated in the embodiments of this application does not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0244] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0245] Among them, the controller may be the nerve center and command center of the electronic device. The controller may generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching instructions and executing instructions.

[0246] A memory can also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store the instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can directly call it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0247] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0248] The charging management module 140 is configured to receive a charging input from a charger. Herein, the charger may be a wireless charger or a wired charger.

[0249] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives the inputs from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160, etc.

[0250] The wireless communication function of the electronic device can be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modulation and demodulation processor, and the baseband processor, etc.

[0251] Antenna 1 and Antenna 2 are used for transmitting and receiving electromagnetic wave signals. Each antenna in the electronic device can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example, Antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0252] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc. applied to the electronic device. The mobile communication module 150 can include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc.

[0253] The modulation and demodulation processor can include a modulator and a demodulator.

[0254] The wireless communication module 160 can provide solutions for wireless communications including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc. applied to the electronic device. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via Antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signals to be sent from the processor 110, perform frequency modulation and amplification on them, and convert them into electromagnetic waves through Antenna 2 for radiation.

[0255] In some embodiments, Antenna 1 of the electronic device is coupled to the mobile communication module 150, and Antenna 2 is coupled to the wireless communication module 160, so that the electronic device can communicate with the network and other devices through wireless communication technologies. The wireless communication technologies can include global system for mobile communications (GSM), general packet radio service (GPRS), etc.

[0256] The electronic device realizes the display function through the GPU, the display screen 194, the application processor, etc. The GPU is a microprocessor for image processing, connecting the display screen 194 and the application processor. The GPU is used to execute mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change the display information.

[0257] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can adopt a liquid crystal display (LCD). The display panel can also adopt an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniLED, a microLED, a micro-OLED, a quantum dot light-emitting diode (QLED), etc. for manufacturing. In some embodiments, the electronic device may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0258] The electronic device can realize the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194, the application processor, etc.

[0259] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and the light passes through the lens and is transmitted to the camera photosensitive element. The optical signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also optimize the noise, brightness, and color of the image through algorithms. The ISP can also optimize parameters such as the exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0260] The camera 193 is used to capture static images or videos. An object generates an optical image through a lens and projects it onto a photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transfers the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard format such as RGB or YUV. In some embodiments, the electronic device may include one or N cameras 193, where N is a positive integer greater than 1.

[0261] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0262] The video codec is used to compress or decompress digital videos. The electronic device can support one or more video codecs. In this way, the electronic device can play or record videos in multiple coding formats, such as: Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0263] The NPU is a neural-network (NN) computing processor. By learning from the structure of biological neural networks, such as the transmission pattern between human brain neurons, it can quickly process the input information and can also continuously self-learn. Through the NPU, applications such as intelligent cognition of the electronic device can be realized, such as: image recognition, face recognition, speech recognition, text understanding, etc.

[0264] The internal memory 121 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM).

[0265] The random access memory may include a static random access memory (SRAM), a dynamic random access memory (DRAM), etc.;

[0266] The non-volatile memory may include a disk storage device, a flash memory.

[0267] Flash memories can be classified into NOR Flash, NAND Flash, 3D NAND Flash, etc. according to their operating principles.

[0268] The random access memory can be directly read and written by the processor 110, and can be used to store the operating system or executable programs (such as machine instructions) of other running programs, and can also be used to store data of users and application programs, etc.

[0269] The non-volatile memory can also store executable programs and data of users and application programs, etc., and can be pre-loaded into the random access memory for direct reading and writing by the processor 110.

[0270] The external memory interface 120 can be used to connect to an external non-volatile memory to expand the storage capacity of the electronic device. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to implement the data storage function. For example, files such as music and videos are saved in the external non-volatile memory.

[0271] The electronic device can implement audio functions through the audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and the application processor, etc. For example, music playback, recording, etc.

[0272] The audio module 170 is used to convert digital audio information into an analog audio signal for output, and is also used to convert analog audio input into digital audio signals.

[0273] The speaker 170A, also known as the "loudspeaker", is used to convert an audio electrical signal into a sound signal. The electronic device can listen to music or hands-free calls through the speaker 170A.

[0274] The receiver 170B, also known as the "earpiece", is used to convert an audio electrical signal into a sound signal. When the electronic device answers a call or a voice message, the receiver 170B can be placed close to the ear to listen to the voice.

[0275] The microphone 170C, also known as the "microphone" or "transmitter", is used to convert a sound signal into an electrical signal.

[0276] The headphone jack 170D is used to connect a wired headphone. The headphone jack 170D can be a USB jack 130, or a 3.5 mm open mobile terminal platform (OMTP) standard jack, or a cellular telecommunications industry association of the USA (CTIA) standard jack.

[0277] The pressure sensor 180A is used to sense a pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can include at least two parallel plates having conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device detects the intensity of the touch operation according to the pressure sensor 180A. The electronic device can also calculate the position of the touch according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: When a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, the instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, the instruction to create a new short message is executed.

[0278] The gyroscope sensor 180B can be used to determine the motion posture of the electronic device.

[0279] The barometric pressure sensor 180C is used to measure the barometric pressure.

[0280] The magnetic sensor 180D includes a Hall sensor.

[0281] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device in various directions (generally three axes).

[0282] The distance sensor 180F is used to measure the distance. The electronic device can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device can use the distance sensor 180F to measure the distance to achieve fast focus.

[0283] The proximity light sensor 180G can include, for example, a light emitting diode (LED) and a light detector, such as a photodiode.

[0284] The ambient light sensor 180L is used to sense the ambient light brightness. The electronic device can adaptively adjust the brightness of the display screen 194 according to the sensed ambient light brightness.

[0285] The fingerprint sensor 180H is used to collect fingerprints. The electronic device can use the collected fingerprint characteristics to achieve fingerprint unlocking, access to application locks, fingerprint photography, fingerprint answering of incoming calls, etc.

[0286] The temperature sensor 180J is used to detect temperature.

[0287] The touch sensor 180K, also known as the "touch panel". The touch sensor 180K can be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also known as a "touch screen". The touch sensor 180K is used to detect touch operations acting thereon or nearby. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device, at a different position from the display screen 194.

[0288] The keys 190 include a power-on key, volume keys, etc.

[0289] The motor 191 can generate a vibration prompt.

[0290] The indicator 192 can be an indicator light, which can be used to indicate the charging state, battery level change, and can also be used to indicate messages, missed calls, notifications, etc.

[0291] The SIM card interface 195 is used to connect the SIM card.

[0292] In the embodiments of the present application, the processor 110 can call computer instructions stored in the internal memory 121 to enable the electronic device to execute the method for processing the content in the pictures in the embodiments of the present application.

[0293] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0294] As used in the foregoing embodiments, depending on the context, the term "when" may be construed to mean "if", "after", "in response to determining", or "in response to detecting". Similarly, depending on the context, the phrase "when determining" or "if (the stated condition or event) is detected" may be construed to mean "if determining", "in response to determining", "when (the stated condition or event) is detected", or "in response to detecting (the stated condition or event)".

[0295] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification and appended claims of the present application, the singular forms "a", "an", "the", "above-mentioned", "said", and "this" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in the present application refers to and encompasses any and all possible combinations of one or more of the listed items.

[0296] The terms "first" and "second" are used only for descriptive purposes and should not be construed as implying or suggesting relative importance or implicitly indicating the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality" is two or more.

[0297] In the foregoing embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid-state drive), etc.

[0298] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by relevant hardware instructed by a computer program. This program can be stored in a computer-readable storage medium. When this program is executed, it can include the processes of the above method embodiments. The foregoing storage media include: various media such as ROM, random access memory (RAM), magnetic disks, or optical discs that can store program codes.

Claims

1. A method for processing content in a picture, characterized in that, the method includes: An electronic device displays a picture; In response to an operation on the picture, the electronic device highlights a first text in the picture and annotates the entity type of the first text at the first text; the entity type of the first text is determined by comparing the entity type to be verified corresponding to the first text with at least one entity type associated with the text in the document card to which the first text belongs; the entity type to be verified is based on the structure of the first text; In response to an operation on the first text, the electronic device provides service options associated with the entity type of the first text.

2. The method according to claim 1, characterized in that, after the electronic device responds to an operation on the picture and before highlighting the first text in the picture located in the document card, the method further includes: The electronic device identifies the document card from the picture and identifies the first text from the picture; Determine the type attribute of the document card and the entity type to be verified of the first text; When it is determined that the first text is located in the document card, the electronic device compares at least one entity type associated with the text in the document card with the entity type to be verified; The electronic device determines that at least one entity type associated with the text in the document card includes the entity type to be verified.

3. The method according to claim 1 or 2, characterized in that, when the document card is a table and the document card further includes a second text, the method further includes: The electronic device highlights the second text; When it is determined that the second text and the first text are not in the same row or the same column in the document card, or the entity types of the first text and the second text are different, when highlighting the second text, the corresponding entity type of the second text is also annotated at the second text.

4. The method according to claim 3, characterized in that, when it is determined that the second text and the first text are in the same row or the same column in the document card and the entity types of the first text and the second text are the same, the corresponding entity type of the second text is not annotated at the second text in the picture.

5. The method according to any one of claims 1-4, characterized in that, in response to an operation on the first text, the electronic device provides service options associated with the entity type of the first text, specifically including: In response to a first operation on the first text, the electronic device displays a first prompt on the first side of the screen, and the first prompt is used to prompt the user that there is a service bar hidden on the first side; In response to an operation of dragging the first text to a first hot zone, the electronic device closes the first prompt and displays the hidden service bar, where at least one service option associated with the entity type of the first text is displayed in the service bar, and the distance from the position farthest from the first side in the first hot zone to the first side is less than or equal to a first distance threshold.

6. The method according to claim 5, wherein, when the first distance threshold is 0, the first hot zone is the first side.

7. The method according to claim 5 or 6, wherein, when the first side is a side edge of the screen of the electronic device and the electronic device displays the hidden service bar, the method further includes: The electronic device shrinks the first text until the height of the first text is less than the height of the service option.

8. The method according to claim 5 or 6, wherein, when the first side is the top edge or the bottom edge of the screen of the electronic device and the electronic device displays the hidden service bar, the method further includes: The electronic device shrinks the first text until the length of the first text is less than the length of the service option.

9. The method according to claim 7 or 8, wherein, the method further includes: When dragging the shrunk first text to the first service option in the service bar and in response to an operation of releasing the shrunk first text, the electronic device activates the first service.

10. The method according to any one of claims 1-9, wherein, in response to an operation on the picture, when the electronic device highlights the first text in the picture that is located in a document card and marks the entity type of the first text at the first text, the method further includes: The electronic device also highlights a first object other than the first text in the picture and marks the object type of the first object at the first object; wherein, the first object includes a third text or a graphic entity that is not in the document card.

11. The method according to claim 10, wherein, when the first object includes a graphic entity, before the electronic device highlights the graphic entity, the method further includes: The electronic device determines that the blur degree of the graphic entity is less than a first preset value, the Euclidean distance between the graphic entity and the center of the picture is less than a second preset value, and the proportion of the graphic entity in the picture is greater than a third preset value.

12. The method according to claim 10 or 11, wherein, when the first object is a graphic entity, the object type of the first object includes a person, a vehicle, a building, an animal, a plant, a document card, and a QR code; when the first object is a text, the object type of the first object includes a mobile phone number, a landline number, an address, a website address, a bank card number, and an express waybill number.

13. An electronic device, wherein, comprising: One or more processors and a memory; the memory is coupled to the one or more processors, and the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the electronic device to execute the method according to any one of claims 1-12.

14. A computer-readable storage medium, comprising computer instructions, wherein, when the computer instructions run on an electronic device, the electronic device is caused to execute the method according to any one of claims 1-12.

15. A chip system, the chip system is applied to an electronic device, wherein, the chip system includes one or more processors, and the processors are used to call computer instructions to cause the electronic device to execute the method according to any one of claims 1-12.

Images