Text point reading method, computer equipment and storage medium
By extracting and comparing the feature information of the text point reading area image and determining the mapping starting point, the problems of low point reading efficiency and poor accuracy in the prior art are solved, and efficient and accurate point reading functions are achieved.
Patent Information
- Application Number
- CN202111666321.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-30
- Publication Date
- 2025-03-28
- Estimated Expiration
- 2041-12-30
AI Technical Summary
Paper books used in assisted learning in prior art refer to low reading function and poor accuracy, especially when the text is curved, poor image stitching effect affects the accuracy of word point reading.
By obtaining the area image corresponding to the actual starting point and the end point of the text point reading, extracting the text feature information and image feature information, comparing it to determine the mapping starting point, and determining the selected text content based on the mapping starting point and the end point to read the point reading process.
Improve the efficiency and accuracy of point reading, avoid the time-consuming image acquisition and stitching, and is not affected by the surface of the text.
Smart Images
Figure CN114359910B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image recognition technology, and in particular to a text point reading method, a computer device and a storage medium. Background Art
[0002] Nowadays, in order to assist learning, the market has launched a finger-reading function for paper books. Users can read out the corresponding audio by pointing their fingers at sentences or words they don't know in the book. The main means of implementation is to capture an image at a certain interval when recognizing that the user's fingertips touch the book, until the user lifts his fingertips, and then stitch the multiple captured images, and recognize the stitched images, so as to realize word-picking and point-reading. Since it is necessary to continuously capture images and stitch them together, it is time-consuming; and for texts with a certain thickness such as teaching materials and textbooks, they are usually in a curved state after opening, rather than a flat surface, which will result in poor image stitching of the multiple captured images, thereby affecting the accuracy of word-picking and point-reading.
[0003] Therefore, how to improve the efficiency and accuracy of point reading becomes a problem that needs to be solved urgently. Summary of the invention
[0004] The present application provides a text point reading method, a computer device and a storage medium, which can improve the efficiency and accuracy of point reading.
[0005] In a first aspect, the present application provides a text point reading method, the method comprising:
[0006] Acquire a first area image corresponding to the actual starting point of text point reading;
[0007] Acquire first text feature information and / or first image feature information corresponding to the first area image;
[0008] Acquire a second area image corresponding to the text reading termination point;
[0009] Acquire second text feature information and / or second image feature information corresponding to the second area image;
[0010] Comparing the first text feature information with the second text feature information, and / or comparing the first image feature information with the second image feature information, to determine that the actual starting point is mapped to a mapping starting point in the second area image;
[0011] Based on the mapping start point and the end point, the text content selected by point reading is determined to perform point reading processing on the text content.
[0012] In a second aspect, the present application further provides a computer device, the computer device comprising:
[0013] Memory and processor;
[0014] Wherein, the memory is connected to the processor and is used to store programs;
[0015] The processor is used to implement the steps of the text point reading method as described in any one of the embodiments of the present application by running the program stored in the memory.
[0016] In a third aspect, the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements the steps of the text reading method as described in any one of the embodiments of the present application.
[0017] The text point reading method, computer device and storage medium disclosed in the present application obtain the first area image corresponding to the actual starting point of the text point reading, obtain the first text feature information and / or the first image feature information corresponding to the first area image, and obtain the second area image corresponding to the end point of the text point reading, obtain the second text feature information and / or the second image feature information corresponding to the second area image, then compare the first text feature information with the second text feature information, and / or compare the first image feature information with the second image feature information, determine the mapping starting point of the actual starting point mapped in the second area image, and determine the text content selected for point reading based on the mapping starting point and the end point, so as to perform point reading processing on the text content. In the whole point reading process, it is not necessary to collect images and perform image splicing processing, thereby improving the efficiency of point reading; and based on the matching of text feature information and / or image feature information, the mapping starting point position is obtained, so as to determine the selected text content for point reading, which is not affected by the curved surface of the text, thereby improving the accuracy of point reading.
[0018] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0020] Figure 1 It is a schematic diagram of the steps of a text point reading method provided by an embodiment of the present application;
[0021] Figure 2is a schematic diagram of a first region image provided by an embodiment of the present application;
[0022] Figure 3 It is a schematic diagram of the steps of another text point reading method provided by an embodiment of the present application;
[0023] Figure 4 is a schematic diagram of a pointing vector provided in an embodiment of the present application;
[0024] Figure 5 It is a schematic block diagram of a computer device provided in an embodiment of the present application.
[0025] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. DETAILED DESCRIPTION
[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0027] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.
[0028] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0029] It should be understood that in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first", "second" and the like are used to distinguish between the same items or similar items with substantially the same functions and effects. For example, the first callback function and the second callback function are only used to distinguish different callback functions, and their order is not limited. Those skilled in the art can understand that the words "first", "second" and the like do not limit the quantity and execution order, and the words "first", "second" and the like do not necessarily limit them to be different.
[0030] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0031] Nowadays, in order to assist learning, the market has launched a finger-reading function for paper books. Users can read out the corresponding audio by pointing their fingers at sentences or words they don't know in the book. The main means of implementation is to capture an image at a certain interval when recognizing that the user's fingertips touch the book, until the user lifts his fingertips, and then stitch the multiple captured images, and recognize the stitched images, so as to realize word-picking and point-reading. Since it is necessary to continuously capture images and stitch them together, it is time-consuming; and for texts with a certain thickness such as teaching materials and textbooks, they are usually in a curved state after opening, rather than a flat surface, which will result in poor image stitching of the multiple captured images, thereby affecting the accuracy of word-picking and point-reading.
[0032] To this end, the embodiments of the present application provide a text point reading method, a computer device and a storage medium, wherein the efficiency and accuracy of point reading can be improved.
[0033] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.
[0034] See also Figure 1 , Figure 1 The present invention provides a flowchart of a text point reading method provided in an embodiment of the present invention. The text point reading method can be applied in a terminal device, wherein the terminal device can be a mobile phone, a tablet computer, a laptop computer, a desktop computer, a personal digital assistant, a wearable device or a smart screen, a smart TV, a projector, a point reading pen, a computer device, etc., but is certainly not limited thereto.
[0035] like Figure 1 As shown, the text point reading method can be applied to a computer device, and the text point reading method applied to a computer device includes steps S101 to S106.
[0036] S101, obtaining a first area image corresponding to an actual starting point of text point reading.
[0037] Exemplarily, the first area image is generated by photographing a camera or other imaging device. Before starting point reading recognition, the camera or other imaging device is adjusted to a fixed position and angle to ensure that the shooting range of the camera or other imaging device can capture the point reading object and the point reading text content. The point reading object includes a user's finger or a point reading tool, such as a point reading pen.
[0038] When the user starts to read the text, the camera captures the image to obtain the area image corresponding to the actual starting point of the text reading. For the convenience of distinguishing the description, the area image corresponding to the actual starting point of the text reading is referred to as the first area image below.
[0039] In some embodiments, obtaining the first area image corresponding to the actual starting point of text point reading may include:
[0040] When it is detected that a point-reading object touches the text for the first time, a first frame of image is captured by a camera device, wherein the point-reading object includes a user's finger or a point-reading tool;
[0041] The first frame image is intercepted to obtain the first region image.
[0042] When it is detected that a finger or other pointing object of a user touches the text, the current frame is recorded, and a first frame image is captured by a camera or other imaging device. Then, the first frame image is intercepted to obtain a first region image.
[0043] Exemplarily, an image is captured above the point-to-read object according to a preset image size to obtain a first area image.
[0044] It should be noted that the preset image size can be flexibly set according to actual conditions and is not specifically limited here. For example, the width of the preset image size is set to the width of the image captured by the camera device, and the height is set to the height of the preset pixels, such as 200 pixels. That is, above the point-reading object such as the user's finger in the first frame image, a first area image with a width of the first frame image width and a height of 200 pixels is intercepted.
[0045] S102: Obtain first text feature information and / or first image feature information corresponding to the first region image.
[0046] Exemplarily, the first region image is subjected to OCR (Optical Character Recognition) recognition to obtain text feature information corresponding to the first region image. For the convenience of distinguishing descriptions, the text feature information corresponding to the first region image is referred to as first text feature information below.
[0047] For example, Figure 2 As shown, the first region image contains the sentence “how old are you?”. By performing OCR recognition on the first region image, first text feature information including: “how”, “old”, “are”, “you”, and “?” is obtained.
[0048] By performing image recognition processing on the first region image, image feature information corresponding to the first region image is obtained. Exemplarily, the image feature information includes at least one of Multi Channel HOG (Histogram of Oriented Gradient) feature information and SIFT (Scale-invariant feature transform) feature information.
[0049] For the convenience of distinguishing descriptions, the image feature information corresponding to the first region image is referred to as first image feature information hereinafter.
[0050] S103: Acquire a second area image corresponding to the text reading end point.
[0051] When the text reading is terminated, the camera device is used to capture an image to obtain a region image corresponding to the text reading termination point. For the convenience of distinguishing descriptions, the region image corresponding to the text reading termination point is referred to as a second region image below.
[0052] In some embodiments, obtaining the second area image corresponding to the text point reading termination point may include:
[0053] When it is detected that the point-to-read object leaves the text and does not touch the text again within a preset time period, a second frame of image is captured at the same viewing angle by the camera device;
[0054] The second frame image is intercepted to obtain the second region image.
[0055] The preset duration can be flexibly set according to actual conditions and is not specifically limited here. For example, the preset duration is set to 500ms.
[0056] For example, when it is detected that the user's finger leaves the text and does not touch the text again within 500ms, the current frame is recorded, and a second frame image is captured at the same viewing angle through a camera or other imaging device. If the user's finger leaves the text and touches the text again within 500ms, the user's finger is left again for more than 500ms, and the point reading is determined to be terminated. At this time, the current frame is recorded to obtain the second frame image. Afterwards, the second frame image is intercepted to obtain the second area image.
[0057] Exemplarily, the width of the second region image is the image width of the second frame image, and the height of the second region image is the height difference between the actual starting point and the ending point plus the height of the first region image.
[0058] Considering that the actual starting point and the ending point may span multiple rows, the height difference m between the actual starting point and the ending point is calculated, and based on the height of the first area image, such as 200 pixels, the height of the second area image is determined to be (m+200).
[0059] S104: Acquire second text feature information and / or second image feature information corresponding to the second region image.
[0060] Based on the method for obtaining the first text feature information and / or the first image feature information, the second text feature information and / or the second image feature information corresponding to the second region image is obtained. Exemplarily, OCR recognition is performed on the second region image to obtain the second text feature information corresponding to the second region image. Image recognition processing is performed on the second region image to obtain the second image feature information corresponding to the second region image.
[0061] S105: Compare the first text feature information with the second text feature information, and / or compare the first image feature information with the second image feature information to determine a mapping start point where the actual starting point is mapped to the second region image.
[0062] That is, by comparing the first text feature information with the second text feature information, it is determined that the actual starting point is mapped to the mapping starting point in the second region image. Alternatively, by comparing the first image feature information with the second image feature information, it is determined that the actual starting point is mapped to the mapping starting point in the second region image. Alternatively, by comparing the first text feature information with the second text feature information, and comparing the first image feature information with the second image feature information, it is determined that the actual starting point is mapped to the mapping starting point in the second region image.
[0063] In some embodiments, Figure 3 As shown, step S103 may include step S107 and step S108, and step S105 may include sub-step S1051 and sub-step S1052.
[0064] S107, determining that the actual starting point corresponds to a first character in the first region image, and determining a first position of the first character in the first region image;
[0065] S108, determining a pointing vector corresponding to the first position and the actual starting point;
[0066] S1051, determining a second character matching the first character in the second region image;
[0067] S1052: Determine the mapping starting point in the second area image to which the actual starting point is mapped according to the second position of the second text in the second area image and the pointing vector.
[0068] Exemplarily, the text closest to the actual starting point in the first region image is determined as the first text. Figure 2 In the first area image shown, the actual starting point of the text point reading corresponds to "how" in the first area image, and the first word is determined to be "how".
[0069] Exemplarily, the center position of the text box corresponding to the first text is determined as the first position corresponding to the first text.
[0070] According to the actual starting point and the first position, a pointing vector corresponding to the first position and the actual starting point is determined. Exemplarily, the vector pointing from the first position to the actual starting point is determined as the pointing vector. Figure 4 As shown, the vector pointing from the center of the "how" text box to the actual starting point of the user's finger is determined as the pointing vector.
[0071] By comparing the first text feature information with the second text feature information, and / or comparing the first image feature information with the second image feature information, the second text matching the first text in the second region image is determined. For example, if the first text is "how", "how" in the second region image is determined.
[0072] In some embodiments, determining the second text matching the first text in the second region image may include:
[0073] Comparing the first character feature information with the second character feature information, and querying whether the second character feature information contains at least one candidate character matching the first character;
[0074] If the second character feature information does not include the candidate character, comparing the first image feature information with the second image feature information to determine the second character matching the first character in the second region image;
[0075] If the second character feature information includes at least one candidate character, the second character is determined from the at least one candidate character.
[0076] The first character feature information is compared with the second character feature information to check whether the second character feature information contains at least one candidate character matching the first character. For example, if the first character is "how", check whether the second character feature information contains at least one "how".
[0077] If the OCR recognition is wrong, the second character feature information may not contain candidate characters that match the first character. At this time, the first image feature information is compared with the second image feature information to determine the second character that matches the first character in the second region image.
[0078] Since there may be multiple identical characters in the first region image / second region image, the second character feature information may include one or more candidate characters matching the first character. At this time, the second character is determined from at least one candidate character. For example, if the second character feature information includes multiple "how", one of the multiple "how" is determined as the second character.
[0079] In some embodiments, determining the second character from at least one of the candidate characters may include:
[0080] Acquire a first ROI image centered on the first character, and acquire a second ROI image centered on each of the candidate characters;
[0081] Calculating a matching degree between the first ROI image and each of the second ROI images;
[0082] If the matching degree between the first ROI image and each of the second ROI images is less than a preset matching degree threshold, comparing the first image feature information with the second image feature information to determine the second text matching the first text in the second region image;
[0083] If the matching degree between the first ROI image and a second ROI image is greater than or equal to the preset matching degree threshold, the candidate text corresponding to the second ROI image is determined as the second text.
[0084] For example, a ROI (region of interest) area is taken with the first character as the center to obtain a first ROI image corresponding to the ROI area, and a corresponding ROI area is taken with each candidate character as the center to obtain a second ROI image corresponding to the ROI area of each candidate character.
[0085] Afterwards, the matching degree between the first ROI image and each second ROI image is calculated.
[0086] Exemplarily, calculating the matching degree between the first ROI image and each of the second ROI images may include:
[0087] Acquire image feature information of the first ROI image and image feature information of each of the second ROI images;
[0088] The image feature information of the first ROI image is compared with the image feature information of each of the second ROI images to determine the matching degree between the first ROI image and each of the second ROI images.
[0089] For example, image recognition processing is performed on the first ROI image to obtain Multi Channel HOG feature information of the first ROI image, and image recognition processing is performed on each second ROI image to obtain Multi Channel HOG feature information of each second ROI image. The Multi Channel HOG feature information of the first ROI image is compared with the Multi Channel HOG feature information of each second ROI image to determine the matching degree between the first ROI image and each second ROI image.
[0090] A preset matching degree threshold corresponding to the image feature information is preset, and the preset matching degree threshold can be flexibly set according to actual conditions, and is not specifically limited here.
[0091] If the matching degree between the first ROI image and each second ROI image is less than a preset matching degree threshold, the first image feature information is compared with the second image feature information to determine the second text that matches the first text in the second region image. If the matching degree between the first ROI image and a second ROI image is greater than or equal to the preset matching degree threshold, the candidate text corresponding to the second ROI image is determined as the second text.
[0092] Exemplarily, the center position of the text box corresponding to the second text is determined as the second position of the second text in the second region image.
[0093] The second position is the mapping position of the first position in the second region image. Therefore, according to the second position and the pointing vector, the mapping starting point where the actual starting point is mapped in the second region image can be determined.
[0094] S106: Determine the text content selected by point reading based on the mapping start point and the end point, so as to perform point reading processing on the text content.
[0095] After obtaining the mapping start point and end point in the second region image, the text content selected by point reading is determined according to the mapping start point and end point.
[0096] In some embodiments, determining the text content selected by point reading based on the mapping start point and the end point may include:
[0097] If the mapping start point and the end point correspond to the same text line, the text content between the mapping start point and the end point in the same text line is determined as the selected text content;
[0098] If the mapping start point and the end point point to different text lines, the text content is determined from a plurality of text lines pointed to by the mapping start point to the end point.
[0099] Since the point reading may start and end in the same line or may cross lines, if the mapping start point and end point point to the same text line, that is, there is no cross-line, the text content from the mapping start point to the end point in the same text line is determined as the text content selected by the point reading. If the mapping start point and end point point to different text lines, that is, there is cross-line, the text content selected by the point reading is determined from the multiple text lines pointed to by the mapping start point to the end point.
[0100] In some embodiments, determining the text content from a plurality of text lines pointed to by the mapping starting point to the end point may include:
[0101] Divide the multiple text lines into paragraphs and perform paragraph center clustering to obtain multiple candidate text lines after clustering;
[0102] Calculating the semantic relevance of two adjacent candidate text lines in order according to the positive order of the plurality of candidate text lines, and determining a first text line set based on the semantic relevance;
[0103] Calculating the semantic relevance of two adjacent candidate text lines in reverse order of the plurality of candidate text lines, and determining a second text line set based on the semantic relevance;
[0104] The union of the first text line set and the second text line set is determined as the text content.
[0105] If the mapping start point and end point correspond to different text lines, it is possible to recognize adjacent paragraphs or the user selects multiple paragraphs during OCR recognition. First, the multiple text lines corresponding to the mapping start point and end point are divided into paragraphs according to the paragraph interval, and the center position of each paragraph is determined. The paragraph centers are clustered based on the center position of each paragraph to obtain multiple candidate text lines after clustering.
[0106] The semantic relevance of two adjacent candidate text lines is calculated in order of the plurality of candidate text lines, and a first text line set is determined based on the semantic relevance. Furthermore, the semantic relevance of two adjacent candidate text lines is calculated in order of the plurality of candidate text lines in reverse order, and a second text line set is determined based on the semantic relevance.
[0107] In some embodiments, determining the first set of text lines based on the semantic relevance may include:
[0108] If the semantic relevance between the first candidate text line and the second candidate text line is greater than or equal to a preset relevance threshold, selecting the second candidate text line and adding it to the first text line set;
[0109] If the semantic relevance between the first candidate text line and the second candidate text line is less than the preset relevance threshold, then the second candidate text line is not selected;
[0110] The first candidate text line is a text line in the first text line set, and the second candidate text line is a text line adjacent to the first candidate text line.
[0111] Exemplarily, a preset correlation threshold corresponding to the semantic correlation is preset, and the preset correlation threshold can be flexibly set according to actual conditions, and is not specifically limited here.
[0112] Starting from the candidate text line corresponding to the mapping start point, add the candidate text line to the first text line set, and calculate the semantic relevance between it and the next candidate text line. If the semantic relevance is greater than or equal to the preset relevance threshold, the next candidate text line is considered to be the selected text line and is added to the first text line set. If the semantic relevance is less than the preset relevance threshold, the next candidate text line is considered not to be the selected text line and is not added to the first text line set. After that, continue to calculate the semantic relevance with the next candidate text line, and so on, until the candidate text line corresponding to the end point is added to the first text line set.
[0113] Starting from the candidate text line corresponding to the end point, add the candidate text line to the second text line set, and calculate the semantic relevance between it and the previous candidate text line. If the semantic relevance is greater than or equal to the preset relevance threshold, the previous candidate text line is considered to be the selected text line and is added to the second text line set. If the semantic relevance is less than the preset relevance threshold, the previous candidate text line is considered not to be the selected text line and is not added to the second text line set. After that, continue to calculate the semantic relevance with the previous candidate text line, and so on, until the candidate text line corresponding to the mapping starting point is added to the second text line set.
[0114] The first text line set and the second text line set are combined, and the text lines in the combined set are arranged in order from the mapping start point to the end point to obtain the selected text line. The text content corresponding to the selected text line is the text content selected by the point reading.
[0115] Afterwards, the selected text content can be processed by point reading, such as reading out the audio corresponding to the text content.
[0116] The text point reading method disclosed in the above embodiment obtains the first area image corresponding to the actual starting point of the text point reading, obtains the first text feature information and / or the first image feature information corresponding to the first area image, and obtains the second area image corresponding to the end point of the text point reading, obtains the second text feature information and / or the second image feature information corresponding to the second area image, and then compares the first text feature information with the second text feature information, and / or compares the first image feature information with the second image feature information, determines the mapping starting point of the actual starting point mapped to the second area image, and determines the text content selected for point reading based on the mapping starting point and the end point, so as to perform point reading processing on the text content. In the entire point reading process, it is not necessary to collect images and perform image splicing processing, thereby improving the efficiency of point reading; and based on the matching of text feature information and / or image feature information, the mapping starting point position is obtained, so as to determine the selected text content for point reading, which is not affected by the curved surface of the text, thereby improving the accuracy of point reading.
[0117] See also Figure 5 , Figure 5 is a schematic block diagram of a computer device provided in an embodiment of the present application. Figure 5 As shown, the computer device 300 includes one or more processors 301 and a memory 302. The processor 301 and the memory 302 are connected via a bus, such as an I2C (Inter-integrated Circuit) bus.
[0118] Among them, one or more processors 301 work individually or collectively to execute the steps of the text point reading method provided in the above embodiment.
[0119] Specifically, the processor 301 may be a micro-controller unit (MCU), a central processing unit (CPU) or a digital signal processor (DSP).
[0120] Specifically, the memory 302 may be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a mobile hard disk.
[0121] The processor 301 is used to run the computer program stored in the memory 302, and implement the steps of the text point reading method provided in the above embodiment when executing the computer program.
[0122] Exemplarily, the processor 301 is used to run the computer program stored in the memory 302, and implement the following steps when executing the computer program:
[0123] Acquire a first area image corresponding to the actual starting point of text point reading;
[0124] Acquire first text feature information and / or first image feature information corresponding to the first area image;
[0125] Acquire a second area image corresponding to the text reading termination point;
[0126] Acquire second text feature information and / or second image feature information corresponding to the second area image;
[0127] Comparing the first text feature information with the second text feature information, and / or comparing the first image feature information with the second image feature information, to determine that the actual starting point is mapped to a mapping starting point in the second area image;
[0128] Based on the mapping start point and the end point, the text content selected by point reading is determined to perform point reading processing on the text content.
[0129] In some embodiments, before acquiring the second area image corresponding to the text point reading termination point, the processor 301 is used to implement:
[0130] Determine that the actual starting point corresponds to a first character in the first region image, and determine a first position of the first character in the first region image;
[0131] Determine a pointing vector corresponding to the first position and the actual starting point;
[0132] When implementing the step of determining the mapping starting point of the actual starting point mapped to the second region image, the processor 301 is configured to implement:
[0133] Determining a second character matching the first character in the second region image;
[0134] The mapping starting point where the actual starting point is mapped to the second area image is determined according to the second position of the second text in the second area image and the pointing vector.
[0135] In some embodiments, when implementing the step of determining the second text matching the first text in the second region image, the processor 301 is configured to implement:
[0136] Comparing the first character feature information with the second character feature information, and querying whether the second character feature information contains at least one candidate character matching the first character;
[0137] If the second character feature information does not include the candidate character, comparing the first image feature information with the second image feature information to determine the second character matching the first character in the second region image;
[0138] If the second character feature information includes at least one candidate character, the second character is determined from the at least one candidate character.
[0139] In some embodiments, when implementing the determining of the second character from at least one candidate character, the processor 301 is configured to implement:
[0140] Acquire a first ROI image centered on the first character, and acquire a second ROI image centered on each of the candidate characters;
[0141] Calculating a matching degree between the first ROI image and each of the second ROI images;
[0142] If the matching degree between the first ROI image and each of the second ROI images is less than a preset matching degree threshold, comparing the first image feature information with the second image feature information to determine the second text matching the first text in the second region image;
[0143] If the matching degree between the first ROI image and a second ROI image is greater than or equal to the preset matching degree threshold, the candidate text corresponding to the second ROI image is determined as the second text.
[0144] In some embodiments, when calculating the matching degree between the first ROI image and each of the second ROI images, the processor 301 is configured to implement:
[0145] Acquire image feature information of the first ROI image and image feature information of each of the second ROI images;
[0146] The image feature information of the first ROI image is compared with the image feature information of each of the second ROI images to determine the matching degree between the first ROI image and each of the second ROI images.
[0147] In some embodiments, when the processor 301 implements the step of acquiring the first area image corresponding to the actual starting point of the text point reading, it is configured to implement:
[0148] When it is detected that a point-reading object touches the text for the first time, a first frame of image is captured by a camera device, wherein the point-reading object includes a user's finger or a point-reading tool;
[0149] Performing image interception on the first frame image to obtain the first region image;
[0150] When the processor 301 implements the step of acquiring the second area image corresponding to the text point reading termination point, the processor 301 is used to implement:
[0151] When it is detected that the point-to-read object leaves the text and does not touch the text again within a preset time period, a second frame of image is captured at the same viewing angle by the camera device;
[0152] The second frame image is intercepted to obtain the second region image.
[0153] In some embodiments, when the processor 301 determines the text content selected by point reading based on the mapping starting point and the ending point, it is used to implement:
[0154] If the mapping start point and the end point correspond to the same text line, the text content between the mapping start point and the end point in the same text line is determined as the selected text content;
[0155] If the mapping start point and the end point point to different text lines, the text content is determined from a plurality of text lines pointed to by the mapping start point to the end point.
[0156] In some embodiments, when determining the text content in the plurality of text lines corresponding to the mapping starting point and the ending point, the processor 301 is configured to implement:
[0157] Divide the multiple text lines into paragraphs and perform paragraph center clustering to obtain multiple candidate text lines after clustering;
[0158] Calculating the semantic relevance of two adjacent candidate text lines in order according to the positive order of the plurality of candidate text lines, and determining a first text line set based on the semantic relevance;
[0159] Calculating the semantic relevance of two adjacent candidate text lines in reverse order of the plurality of candidate text lines, and determining a second text line set based on the semantic relevance;
[0160] The union of the first text line set and the second text line set is determined as the text content.
[0161] In some embodiments, when implementing the determining of the first text line set based on the semantic relevance, the processor 301 is configured to implement:
[0162] If the semantic relevance between the first candidate text line and the second candidate text line is greater than or equal to a preset relevance threshold, selecting the second candidate text line and adding it to the first text line set;
[0163] If the semantic relevance between the first candidate text line and the second candidate text line is less than the preset relevance threshold, then the second candidate text line is not selected;
[0164] The first candidate text line is a text line in the first text line set, and the second candidate text line is a text line adjacent to the first candidate text line.
[0165] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements the steps of the text point reading method provided in the above embodiment.
[0166] The computer-readable storage medium may be an internal storage unit of the computer device described in any of the above embodiments, such as a hard disk or memory of the terminal device. The computer-readable storage medium may also be an external storage device of the terminal device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the terminal device.
[0167] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A text point reading method, characterized in that: The method comprises: Acquire a first area image corresponding to the actual starting point of text point reading; Acquire first text feature information and / or first image feature information corresponding to the first area image; Acquire a second area image corresponding to the text reading termination point; Acquire second text feature information and / or second image feature information corresponding to the second area image; Comparing the first text feature information with the second text feature information, and / or comparing the first image feature information with the second image feature information, to determine that the actual starting point is mapped to a mapping starting point in the second area image; Based on the mapping start point and the end point, determine the text content selected by point reading to perform point reading processing on the text content; The step of obtaining a first area image corresponding to the actual starting point of the text point reading comprises: When it is detected that a point-reading object touches the text for the first time, a first frame of image is captured by a camera device, wherein the point-reading object includes a user's finger or a point-reading tool; Performing image interception on the first frame image to obtain the first region image; The step of obtaining a second area image corresponding to the text point reading termination point includes: When it is detected that the point-to-read object leaves the text and does not touch the text again within a preset time period, a second frame of image is captured at the same viewing angle by the camera device; The second frame image is intercepted to obtain the second region image.
2. The method according to claim 1, characterized in that Before obtaining the second area image corresponding to the text point reading termination point, the method includes: Determine that the actual starting point corresponds to a first character in the first region image, and determine a first position of the first character in the first region image; Determine a pointing vector corresponding to the first position and the actual starting point; The determining that the actual starting point is mapped to a mapping starting point in the second area image includes: Determining a second character matching the first character in the second region image; The mapping starting point where the actual starting point is mapped to the second area image is determined according to the second position of the second text in the second area image and the pointing vector.
3. The method according to claim 2, characterized in that The determining of the second text matching the first text in the second region image includes: Comparing the first character feature information with the second character feature information, and querying whether the second character feature information contains at least one candidate character matching the first character; If the second character feature information does not include the candidate character, comparing the first image feature information with the second image feature information to determine the second character matching the first character in the second region image; If the second character feature information includes at least one candidate character, the second character is determined from the at least one candidate character.
4. The method according to claim 3, characterized in that The determining the second character from at least one of the candidate characters comprises: Acquire a first ROI image centered on the first character, and acquire a second ROI image centered on each of the candidate characters; Calculating a matching degree between the first ROI image and each of the second ROI images; If the matching degree between the first ROI image and each of the second ROI images is less than a preset matching degree threshold, comparing the first image feature information with the second image feature information to determine the second text matching the first text in the second region image; If the matching degree between the first ROI image and a second ROI image is greater than or equal to the preset matching degree threshold, the candidate text corresponding to the second ROI image is determined as the second text.
5. The method according to claim 4, characterized in that The calculating the matching degree between the first ROI image and each of the second ROI images includes: Acquire image feature information of the first ROI image and image feature information of each of the second ROI images; The image feature information of the first ROI image is compared with the image feature information of each of the second ROI images to determine the matching degree between the first ROI image and each of the second ROI images.
6. The method according to any one of claims 1 to 5, characterized in that The step of determining the text content selected by point reading based on the mapping starting point and the end point includes: If the mapping start point and the end point correspond to the same text line, the text content between the mapping start point and the end point in the same text line is determined as the selected text content; If the mapping start point and the end point point to different text lines, the text content is determined from a plurality of text lines pointed to by the mapping start point to the end point.
7. The method according to claim 6, characterized in that Determining the text content from the plurality of text lines pointed to by the mapping starting point to the corresponding ending point includes: Divide the multiple text lines into paragraphs and perform paragraph center clustering to obtain multiple candidate text lines after clustering; Calculating the semantic relevance of two adjacent candidate text lines in order according to the positive order of the plurality of candidate text lines, and determining a first text line set based on the semantic relevance; Calculating the semantic relevance of two adjacent candidate text lines in reverse order of the plurality of candidate text lines, and determining a second text line set based on the semantic relevance; The union of the first text line set and the second text line set is determined as the text content.
8. The method according to claim 7, characterized in that The determining a first text line set based on the semantic relevance comprises: If the semantic relevance between the first candidate text line and the second candidate text line is greater than or equal to a preset relevance threshold, selecting the second candidate text line and adding it to the first text line set; If the semantic relevance between the first candidate text line and the second candidate text line is less than the preset relevance threshold, then the second candidate text line is not selected; The first candidate text line is a text line in the first text line set, and the second candidate text line is a text line adjacent to the first candidate text line.
9. A computer device, characterized in that: The computer device comprises: Memory and processor; Wherein, the memory is connected to the processor and is used to store programs; The processor is used to implement the steps of the text point reading method as described in any one of claims 1 to 8 by running the program stored in the memory.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, enables the processor to implement the steps of the text point reading method according to any one of claims 1 to 8.
Citation Information
Patent Citations
Movement control method and device, computer equipment and service robot
CN107341442A
Intelligent reading learning method
CN110569818A