OCR recognition method based on XR glasses, OCR recognition device and XR glasses

Abstract

The present application relates to an OCR recognition method, an OCR recognition device and XR glasses based on XR glasses, which collects a current frame image through an outward-facing camera; extracts similar point pairs between the current frame image and the previous frame image, and determines whether the displacement between the similar point pairs is not less than a preset displacement threshold; when the determination is yes, globally exposes the current frame image based on the exposure parameters of the previous frame image to obtain a first exposure image; obtains the shooting angle of the current frame image, corrects the first exposure image based on the shooting angle, extracts a text box based on the corrected first exposure image, and locally exposes the first exposure image based on the text box to obtain a second exposure image; recognizes characters in the text box in the second exposure image, and generates a display object based on the recognized characters; restores the display angle of the text box according to the shooting angle, and displays the display object on the display screen of the XR glasses according to the display angle; and can improve the accuracy of character recognition in mobile scenarios.

Description

Technical Field

[0001] The present application relates to the field of XR glasses, and in particular to an OCR recognition method, an OCR recognition device and XR glasses based on XR glasses. Background Art

[0002] Optical Character Recognition (OCR) is a technology that converts various document types (such as scanned paper documents, PDF files, or text in images) into text that computers can process. Through OCR, computers can "read" characters in images and convert them into editable and searchable data.

[0003] XR glasses, or Extended Reality (XR) glasses, enhance or extend the user's real-world experience by overlaying digital information on their field of view or creating a fully immersive virtual environment. By embedding OCR functionality into XR glasses, users can view surrounding images through the glasses and then utilize the built-in OCR function to recognize characters within the images.

[0004] Currently, XR glasses with OCR recognition capabilities are primarily used in static scenarios, such as language learning, work assistance, education, and document processing. Character recognition accuracy in dynamic scenarios needs to be improved. For example, when users are traveling and identifying outdoor signs while wearing XR glasses, images can become blurry and vertical text can tilt, ultimately reducing character recognition accuracy. Therefore, the low accuracy of OCR recognition in mobile scenarios has become a difficult technical bottleneck to overcome. Summary of the Invention

[0005] Current XR glasses-based OCR recognition methods suffer from low character recognition accuracy in mobile scenarios. Therefore, it is necessary to address this technical issue by providing an XR glasses-based OCR recognition method, OCR recognition device, and XR glasses that can improve character recognition accuracy in mobile scenarios.

[0006] In a first aspect, the present application provides an OCR recognition method based on XR glasses. The method comprises:

[0007] Capture the current frame image through the outward-facing camera of the XR glasses;

[0008] Extracting similar point pairs between the current frame image and the previous frame image, and determining whether the displacement between the similar point pairs is not less than a preset displacement threshold;

[0009] When it is determined that the displacement between the pair of similar points is not less than the preset displacement threshold, globally exposing the current frame image based on the exposure parameter of the previous frame image to obtain a first exposure image;

[0010] obtaining a shooting angle of the outward-facing camera when capturing the current frame image, correcting the first exposure image based on the shooting angle, extracting a text frame based on the corrected first exposure image, and partially exposing the first exposure image based on the text frame to obtain a second exposure image;

[0011] recognizing characters in the text box in the second exposure image, and generating a display object based on the recognized characters;

[0012] The display angle of the text box is restored according to the shooting angle, and the display object is displayed on the display screen of the XR glasses according to the display angle.

[0013] In one embodiment, extracting similar point pairs between the current frame image and the previous frame image, and determining whether the displacement between the similar point pairs is not less than a preset displacement threshold, includes:

[0014] Taking the previous frame image as a reference object, performing posture correction on the current frame image;

[0015] respectively extracting similar point pairs between the corrected current frame image and the previous frame image, and calculating the horizontal or vertical displacement of the similar point pairs between the corrected current frame image and the previous frame image;

[0016] Among multiple pairs of similar point pair displacements, remove the maximum displacement and the minimum displacement, calculate the average of the remaining similar point pair displacements, and obtain the target similar point pair displacement;

[0017] It is determined whether the displacement of the target similar point pair is not less than the preset displacement threshold.

[0018] In one embodiment, obtaining a shooting angle of the outward-facing camera when capturing the current frame image, and correcting the first exposure image based on the shooting angle includes:

[0019] Calculating the yaw angle of the inertial measurement unit (IMU) in the XR glasses, and obtaining the shooting angle of the outward-facing camera based on the IMU.

[0020] The first exposure image is offset along the Y direction based on the shooting angle to correct the tilt of the first exposure image.

[0021] In one embodiment, performing partial exposure on the first exposure image based on the text box includes:

[0022] Taking the pixel area corresponding to the text box as the target pixel area, calculating the brightness histogram of the target pixel area;

[0023] The exposure gain of the target pixel area is increased according to the brightness histogram to obtain the second exposure image.

[0024] In one embodiment, recognizing characters in the text box in the second exposure image and generating a display object based on the recognized characters includes:

[0025] Acquire the original text of the characters recognized based on the previous frame of image to obtain a first character string;

[0026] Recognize the original characters in the text box to obtain a second character string;

[0027] Comparing the first character string and the second character string for similarity to obtain a character string similarity;

[0028] Determining whether the similarity of the character strings is not less than a preset similarity threshold;

[0029] If it is determined that the string similarity is not lower than the preset similarity threshold, reading the translation text of the first string from the cache list and using the translation text of the first string as the translation text of the second string;

[0030] The translated text of the second character string is used as the display object.

[0031] In one embodiment, performing a similarity comparison on the first character string and the second character string to obtain character string similarity includes:

[0032] Calculating an edit distance between the first character string and the second character string;

[0033] The character string similarity is determined according to the edit distance.

[0034] In one embodiment, before displaying the display object on the display screen of the XR glasses according to the display angle, the method further includes:

[0035] The left eye image and the right eye image of the user are captured by the left and right binocular inward-facing cameras of the XR glasses;

[0036] identifying a left eye pupil in the left eye image and identifying a right eye pupil in the right eye image;

[0037] Obtaining the size of the display screen and the distance between the inward-facing camera and the display screen, and converting the intersection of the sight lines of the left pupil and the right pupil into a sight line landing point on the display screen based on the size of the display screen and the distance between the inward-facing camera and the display screen;

[0038] The display object is displayed at the sight point of the display screen according to the display angle.

[0039] In one embodiment, identifying the left eye pupil in the left eye image and identifying the right eye pupil in the right eye image includes:

[0040] performing threshold segmentation on the left-eye image and the right-eye image respectively;

[0041] Counting target pixels in the left-eye image and the right-eye image after threshold segmentation, calculating the pupil area according to the counted number of target pixels, and calculating the pupil radius according to the pupil area;

[0042] Setting a window with the upper left corner pixel of the left-eye image and the right-eye image as a starting point and the radius as a side length;

[0043] Sliding the window in the left-eye image and the right-eye image respectively, and detecting the number of black pixels in the window during each sliding;

[0044] The left eye pupil corresponding to the left eye image and the right eye pupil corresponding to the right eye image are determined according to the area that slides to the maximum target pixel number.

[0045] In a second aspect, the present application provides an OCR recognition device, comprising:

[0046] The image acquisition module is used to capture the current frame image through the outward-facing camera of the XR glasses;

[0047] a first exposure module configured to extract similar point pairs between the current frame image and the previous frame image, and determine whether a displacement between the similar point pairs is not less than a preset displacement threshold; and when it is determined that the displacement between the similar point pairs is not less than the preset displacement threshold, perform global exposure on the current frame image based on the exposure parameters of the previous frame image to obtain a first exposure image;

[0048] a second exposure module, configured to obtain a shooting angle of the outward-facing camera when capturing the current frame image, correct the first exposure image based on the shooting angle, extract a text frame based on the corrected first exposure image, and partially expose the first exposure image based on the text frame to obtain a second exposure image;

[0049] a recognition module, configured to recognize characters in the text box in the second exposure image, and generate a display object based on the recognized characters;

[0050] The display module is used to restore the display angle of the text box according to the shooting angle, and display the display object on the display screen of the XR glasses according to the display angle.

[0051] In a third aspect, the present application further provides an XR glasses comprising: a frame, and an outward-facing camera, a display screen, and a control module installed in the frame; wherein,

[0052] The outward-facing camera and the display screen are respectively connected to the control module;

[0053] The control module is used to execute the steps of the method described in the first aspect above.

[0054] The aforementioned XR glasses-based OCR recognition method, OCR recognition device, and XR glasses take into account the generally low clarity of images captured in mobile scenarios, which reduces character recognition accuracy. Furthermore, some images containing vertically aligned text are easily affected by the shooting angle of the outward-facing camera, which also reduces character recognition accuracy. To address these issues, the method calculates the displacement of similar point pairs between the previous and next frames. When the displacement is too large, global and local exposure are triggered to improve image clarity. The image is then corrected based on the shooting angle to straighten tilted text boxes, ultimately improving character recognition accuracy in mobile scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 Schematic diagram of the architecture of XR glasses in one embodiment;

[0056] Figure 2 A schematic diagram of the architecture of XR glasses in another embodiment;

[0057] Figure 3 Schematic diagram of an application environment of XR glasses in one embodiment;

[0058] Figure 4 Flowchart of an OCR recognition method based on XR glasses in one embodiment;

[0059] Figure 5 is a flowchart of a translation method based on character similarity comparison in one embodiment;

[0060] Figure 6 is a flow chart of a display method based on eye tracking in one embodiment;

[0061] Figure 7 A schematic diagram of perspective projection matrix calculation in one embodiment;

[0062] Figure 8 A schematic diagram of a text box projection position in one embodiment;

[0063] Figure 9 FIG. 1 is a schematic structural diagram of an OCR recognition device in an embodiment. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0065] Unless otherwise defined, technical or scientific terms used in this application shall have the ordinary meanings as understood by persons of ordinary skill in the art to which this application belongs. The terms "a," "an," "the," "these," and similar expressions in this application do not denote limitations on quantity and may be singular or plural. The terms "comprise," "include," "have," and any variations thereof, as used in this application, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include unlisted steps or modules (units) or other steps or modules (units) inherent to the process, method, product, or device. The terms "connected," "connected," "coupled," and similar expressions used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. As used in this application, "plurality" means two or more. "And / or" describes an association between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone; A and B exist simultaneously; or B exists alone. Generally, the character " / " indicates that the objects in the preceding and following relationship are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0066] In some embodiments, Figure 1 A schematic diagram of the architecture of XR glasses is provided, such as Figure 1 As shown, the XR glasses include: a frame, and an outward-facing camera 11, a display screen 12 and a control module 13 installed in the frame; the outward-facing camera 11 and the display screen 12 are respectively connected to the control module 13.

[0067] XR glasses can be wearable devices that integrate any of the following technologies: virtual reality (VR), augmented reality (AR), and mixed reality (MR). MR glasses are an advanced form of augmented reality (AR) technology that can overlay digital information onto the real environment and enable interaction, allowing users to see the real environment around them while also displaying digital information and objects.

[0068] In this embodiment, the outward-facing camera 11 is mounted on the outside of the frame and is used to capture images of external scenes, including but not limited to outdoor signs (such as billboards, road signs, subway station name displays, etc.), supermarket price tags, restaurant menus, shopping mall floor guide maps, elevator button signs, etc. The display screen 12 is used to display the display objects output by the control module 13. The control module 13 is used to execute the OCR recognition method based on XR glasses. The description of this method can be found in the following Figure 4 The process shown.

[0069] In some embodiments, Figure 2 Another schematic diagram of the architecture of XR glasses is provided, such as Figure 2 As shown, the XR glasses may further include an inward-facing camera 14, which is installed on the inner side of the frame. The inward-facing camera 14 includes left and right binocular cameras, which are respectively used to capture the left eye image and the right eye image of the user (i.e., the wearer), and input the left eye image and the right eye image to the control module 13. The control module 13 implements eye tracking of the user based on the captured left eye image and the right eye image to determine the sight point of the displayed object on the display screen 12, thereby simulating the real effect of the user viewing the real object.

[0070] In some embodiments, the XR glasses also include an inertial measurement unit (IMU) for calculating the shooting angle of the outward-facing camera 11. This includes an accelerometer and a magnetometer. The accelerometer is used to collect the current acceleration of the IMU on the X, Y, and Z axes, while the magnetometer is used to collect the current orientation of the IMU.

[0071] In some embodiments, Figure 3 A schematic diagram of the application environment of XR glasses is provided, such as Figure 3As shown, the XR glasses 1 are connected to the cloud server 2 via a communication network. The data storage system 3 can store data that the cloud server 2 needs to process, such as a language library. The data storage system 3 can be integrated on the cloud server 2, or placed on other clouds or other network servers. The cloud server 2 can be implemented as an independent server or a server cluster consisting of multiple servers. After the XR glasses 1 recognize characters based on the captured scene image, the XR glasses 1 can request the cloud server 2 to translate the recognized characters and return the translated text. Finally, the XR glasses 1 will display the translated text on the display screen.

[0072] In some embodiments, Figure 4 A flowchart of an OCR recognition method based on XR glasses is provided. Figure 4 As shown, the process includes the following steps:

[0073] Step S101: Capture the current frame image through the outward-facing camera of the XR glasses.

[0074] The outward-facing camera is installed on the outside of the frame and is used to capture images of external scenes, including but not limited to outdoor signs (such as billboards, road signs, subway station name displays, etc.), supermarket price tags, restaurant menus, shopping mall floor guide maps, elevator button signs, etc.

[0075] Step S102 : extracting similar point pairs between the current frame image and the previous frame image, and determining whether the displacement between the similar point pairs is not less than a preset displacement threshold.

[0076] In this step, the posture of the current frame image is corrected with the previous frame image as a reference object; the similar point pairs of the corrected current frame image and the previous frame image are extracted respectively, and the displacement of the similar point pairs in the horizontal or vertical direction between the corrected current frame image and the previous frame image is calculated; among the multiple pairs of similar point pair displacements, the maximum displacement and the minimum displacement are removed, and the average value of the remaining similar point pair displacements is calculated to obtain the target similar point pair displacement; and it is determined whether the target similar point pair displacement is not lower than the preset displacement threshold.

[0077] Among them, when calculating the displacement of the target similar point pairs, by removing the maximum displacement and the minimum displacement and obtaining the average value of the remaining similar point pairs' displacements, it can play a role in resisting the interference of outliers, so that the target similar point pairs' displacements finally calculated are representative.

[0078] For example, the preset displacement threshold can be 1 / 4 of the side length of the image (the current frame or the previous frame) in the X or Y direction. The difference Δx in the X direction and the difference Δy in the Y direction of the similar points in the two images can be calculated. After removing the maximum and minimum differences, the average is taken to obtain the displacement of the target similar point pair. Assuming that the side length of the image in the X direction is w and the side length in the Y direction is h, determine whether Δx ≥ w / 4 or Δy ≥ h / 4 holds.

[0079] This step calculates the displacement of similar point pairs between the two frames of images, and determines whether the two frames of images correspond to the same scene based on the displacement of similar point pairs.

[0080] Step S103 : When it is determined that the displacement between the similar point pairs is not less than the preset displacement threshold, the current frame image is globally exposed based on the exposure parameters of the previous frame image to obtain a first exposure image.

[0081] If any of the conditions Δx ≥ w / 4 and Δy ≥ h / 4 hold, the displacement between the similar point pairs is determined to be no less than the preset displacement threshold, indicating a significant scene change between the previous and next frames. The current frame may not be able to use the character recognition results from the previous frame, requiring a new round of character recognition. Considering that images captured in moving scenes generally have lower clarity, which affects character recognition accuracy, it is necessary to first globally expose the current frame based on the exposure parameters of the previous frame to improve its overall clarity.

[0082] When performing global exposure on the current frame image based on the exposure parameters of the previous frame image, you can extract the exposure parameters of the previous frame image (such as aperture value, shutter speed, ISO sensitivity, etc.) and adjust the exposure value, contrast, highlights, shadows, etc. of the current frame image according to the exposure parameters.

[0083] Step S104, obtaining the shooting angle of the outward-facing camera when capturing the current frame image, correcting the first exposure image based on the shooting angle, extracting the text box based on the corrected first exposure image, and partially exposing the first exposure image based on the text box to obtain a second exposure image.

[0084] Considering that in real-world scenarios, some signs have vertically aligned text, when capturing images containing vertically aligned text, the outward-facing camera's shooting angle deviates from the horizontal reference line due to changes in the user's viewing perspective (e.g., looking down or up), causing the captured text to be tilted, thereby affecting character recognition accuracy. Therefore, this step corrects the first exposure image based on the outward-facing camera's shooting angle to reduce the impact of text tilt on character recognition. The shooting angle refers to the vertical tilt between the outward-facing camera and the center of the subject, for example, 30° when looking down and -30° when looking up.

[0085] In this step, the yaw angle can be calculated by the inertial measurement unit in the XR glasses, and the shooting angle of the outward-facing camera can be obtained based on the yaw angle; based on the shooting angle, the first exposure image is staggered along the Y direction to correct the tilt of the first exposure image.

[0086] The inertial measurement unit includes an accelerometer and a magnetometer. The accelerometer is used to collect the current acceleration of the IMU on the X, Y, and Z axes, while the magnetometer is used to collect the current orientation of the IMU. The shooting angle of the outward-facing camera is calculated as follows:

[0087] Step 1: Calculate the roll angle and pitch angle based on the accelerometer readings (three-axis accelerometer values). The calculation formula is as follows:

[0088] ;

[0089] Explanation of symbols:

[0090] accelX, accelY, accelZ: three-axis accelerometer values;

[0091] atan2(y,x): four-quadrant inverse tangent function;

[0092] 180 / π: radians to degrees.

[0093] Step 2: Based on the magnetometer reading and the roll and pitch calculated in step 1 above, calculate the yaw angle, which is the shooting angle. The calculation formula is as follows:

[0094] ;

[0095] ;

[0096] Explanation of symbols:

[0097] magReadX, magReadY, magReadZ: magnetometer readings;

[0098] magx: correction magnetic field in the X direction;

[0099] magy: Correction magnetic field in the Y direction.

[0100] After the shooting angle is calculated, the first exposure image is offset along the Y direction based on the shooting angle to correct the tilt of the first exposure image. The calculation formula is as follows:

[0101] ;

[0102] Explanation of symbols:

[0103] x, y: pixel coordinates of the first exposure image without correction;

[0104] x', y': pixel coordinates of the first exposure image after correction;

[0105] dx, dy: the shear amount of the first exposure image in the X and Y directions.

[0106] When the original pixel coordinates (x, y) are known, let dx = 0, dy = tan(yaw), that is, by shearing along the y direction, the pixel coordinates (x', y') of the corrected image can be calculated to achieve correction of the first exposure image.

[0107] The text box is used to find the positions of all text lines in the image. The text box can be extracted using edge detection (such as the Canny operator) and connected domain analysis, or a deep learning method can be used to extract the text box, which is not limited in this embodiment.

[0108] When extracting text frames based on the corrected first exposure image, a text frame set containing at least one candidate text frame is extracted. For example, a billboard image might contain both a store name and address. When generating text frames for the billboard image, the resulting text frames might include text frame A containing both the store name and address, text frame B containing only the store name, and text frame C containing only the store address. These multiple text frames form a text frame set. Within the text frame set, the largest text frame is searched for. A partial exposure of the first exposure image is performed based on this largest text frame to produce a second exposure image.

[0109] By finding the largest text box, the characters to be recognized are as complete as possible. When performing a partial exposure on the first exposure image based on the text box, the pixel area corresponding to the text box is used as the target pixel area, and a brightness histogram of the target pixel area is calculated. The exposure gain of the target pixel area is increased based on the brightness histogram to produce the second exposure image. This partial exposure step, based on the previously described global exposure, precisely enhances the clarity of the area to be recognized, helping to improve character recognition accuracy.

[0110] Step S105 : Recognize characters in the text box in the second exposure image, and generate a display object based on the recognized characters.

[0111] When recognizing characters in the text box in the second exposure image, the text box can be segmented first and then single characters can be recognized. Alternatively, an end-to-end model (such as CNN+RNN+CTC) can be used to directly output the character string.

[0112] When determining the display object based on the recognized characters, the recognized characters themselves may be used as the display object, or the recognized characters may be translated first and then the translated text may be used as the display object.

[0113] Step S106: restore the display angle of the text box according to the shooting angle, and display the display object on the display screen of the XR glasses according to the display angle.

[0114] According to the image shearing formula of the aforementioned embodiment, the display angle of the text box can be restored. The calculation formula is as follows:

[0115] ;

[0116] Explanation of symbols:

[0117] x, y: pixel coordinates of the corrected and restored text box in the second exposure image;

[0118] x', y': pixel coordinates of the text box before correction and restoration in the second exposure image;

[0119] dx, dy: the shear amount of the second exposure image in the X and Y directions.

[0120] When the current pixel coordinates (x', y') are known, let dx=0, dy=tan(yaw). That is, by shearing along the y direction, the pixel coordinates (x, y) of the corrected and restored text box can be calculated to restore the angle of the text box.

[0121] In steps S101 to S106 above, it is considered that the image clarity obtained in mobile scenes is generally not high enough, which will reduce the accuracy of character recognition. In addition, some images containing vertically arranged text are easily affected by the shooting angle of the outward-facing camera, which will also reduce the accuracy of character recognition. In view of the above problems, this embodiment improves image clarity by calculating the displacement of similar point pairs between the previous and next frame images, triggering global exposure and local exposure when the displacement of similar point pairs is too large, and then correcting the image by combining the shooting angle to straighten the tilted text box, ultimately improving the accuracy of character recognition in mobile scenes.

[0122] In some embodiments, when determining a display object based on the recognized characters, the recognized characters are first translated, and then the translated text is used as the display object. When translating the recognized characters, a translation request can be made to a cloud server through a translation interface, and the result returned by the cloud server is used as the display object.

[0123] However, the above translation method has delays and is easily affected by network signals. To solve this problem, Figure 5 A flowchart of a translation method based on character similarity comparison is provided. Figure 5 As shown, recognizing characters in a text box in a second exposure image and generating a display object based on the recognized characters includes the following steps:

[0124] Step S201 : obtaining the original text of characters recognized based on the previous frame of image to obtain a first character string.

[0125] Step S202: Identify the original characters in the text box to obtain a second character string.

[0126] Step S203 : performing a similarity comparison between the first character string and the second character string to obtain character string similarity.

[0127] In this step, the string similarity can be determined by calculating the edit distance between the first string and the second string. The following is a sample code for calculating the edit distance:

[0128] for (int i=1; i <m+1;i++) {

[0129] for (int j=1; j <n+1;j++) {

[0130] / / Perform an insert operation on the basis that the first i characters of the first string are exactly the same as the first j-1 characters of the second string;

[0131] int countByInsert=lev[i][j-1]+1;

[0132] / / Perform a delete operation on the basis that the first i-1 characters of the first string are exactly the same as the first j characters of the second string;

[0133] int countByDel=lev[i-1][j]+1;

[0134] / / Perform a replacement operation based on the fact that the first i-1 characters of the first string are exactly the same as the first j-1 characters of the second string;

[0135] intcountByReplace=word1.charAt(i-1)==word2.charAt(j-1)?

[0136] lev[i-1][j-1]:lev[i-1][j-1]+1;

[0137] / / Calculate the Levenshtein distance between the first i characters of the first string and the first j characters of the second string;

[0138] lev[i][j]=min(countByInsert, countByDel, countByReplace);

[0139] }

[0140] }

[0141] The above example code initializes a two-dimensional array lev[m+1][n+1], where m and n are the lengths of the first and second strings respectively, and lev[i][j] represents the edit distance between the first i characters of the first string and the first j characters of the second string. For each character position i (1≤i≤m) and j (1≤j≤n), the following operations are performed:

[0142] If the i-th character of the first string is equal to the j-th character of the second string, then lev[i][j]=lev[i-1][j-1];

[0143] Otherwise, lev[i][j] takes the minimum value of the following three operations plus 1:

[0144] (1) Insertion operation: lev[i][j-1]+1;

[0145] (2) Deletion operation: lev[i-1][j]+1;

[0146] (3) Replacement operation: lev[i-1][j-1]+1;

[0147] The final edit distance is lev[m][n].

[0148] Step S204: determine whether the character string similarity is not lower than a preset similarity threshold.

[0149] Step S205 : If it is determined that the string similarity is not lower than the preset similarity threshold, the translation text of the first string is read from the cache list, and the translation text of the first string is used as the translation text of the second string.

[0150] If it is determined that the string similarity is not lower than the preset similarity threshold, the first string and the second string are considered the same string. The translation text of the first string can be used for the second string. For the current frame image, the text box size of the previous frame image and the position of the text box of the current frame image can be continued to be used. Exemplarily, the preset similarity threshold can be set to 70%. If the string similarity is not lower than 70%, the first string and the second string are considered the same string.

[0151] Optionally, the characters of the second string can be compared with the characters in the fixed character list first. The fixed character list stores fixed translation information, such as {"Company Introduction", "公司介绍"}, and the translation content can be directly output. For the remaining characters of the second string, the similarity with the cache list of the first string is calculated again. If the similarity of the remaining string is not lower than 70%, it is considered that the characters of the second string already exist in the cache list, and the translation text in the cache list can be directly used. If the similarity of the remaining string is lower than 70%, it is considered that the characters of the second string do not exist in the cache list and need to be re-translated. Re-translation can request translation from the cloud server through the translation interface, and then use the result returned by the cloud server as the translation text of the second string and store the translation text of the second string in the local cache list. Optionally, the cache list can be set with a maximum cache capacity. For example, the maximum string length of the cache is 10 bytes, and the earliest cached characters will be deleted if it exceeds 10 bytes.

[0152] Step S206, use the translation text of the second string as the display object.

[0153] The translation text as the display object will move slowly on the display screen according to the preset refresh frequency.

[0154] In the above steps S201 to S206, considering that it has been determined in the previous step S103 that the displacement between the similar point pairs is not lower than the preset displacement threshold, that is, there is a large displacement between the front and rear frame images, the pose of the XR glasses changes greatly, and it is very likely that the content between the front and rear frame images changes, so the recognized characters need to be re-translated. However, re-translation has the problems of delay and being vulnerable to network signals. Therefore, it is necessary to judge whether the content between the front and rear frame images really changes to decide whether to continue to use the historical translation text to improve the translation efficiency. Therefore, the similarity between the first string and the second string needs to be compared. In this embodiment, by entering the similarity comparison of the strings when the user has a certain pose change (a large displacement between the front and rear frame images), and judging whether to trigger re-translation according to the string similarity, the translation efficiency is improved.

[0155] In some embodiments, Figure 6A flow chart of a display method based on eye tracking is provided, such as Figure 6 As shown, before displaying the display object on the display screen of the XR glasses according to the display angle, the method further includes the following steps:

[0156] Step S301: Use the left and right binocular inward-facing cameras of the XR glasses to capture the user's left eye image and right eye image.

[0157] Step S302: Identify the left eye pupil in the left eye image, and identify the right eye pupil in the right eye image.

[0158] In this step, threshold segmentation is performed on the left eye image and the right eye image respectively; target pixels (such as black pixels, brown pixels, blue pixels or pixels of other pupil colors) are counted in the left eye image and the right eye image after threshold segmentation respectively, the pupil area is calculated according to the number of counted target pixels, and the pupil radius R is calculated according to the pupil area; a window is set with the upper left corner pixel of the left eye image and the right eye image as the starting point (such as (0, 0)) and the radius R as the side length; the window is slid in the left eye image and the right eye image respectively, and the number of black pixels in the window is detected each time the window is slid; the left eye pupil corresponding to the left eye image and the right eye pupil corresponding to the right eye image are determined according to the area that slides to the maximum number of target pixels.

[0159] Step S303, obtain the size of the display screen and the distance between the inward-facing camera and the display screen, and convert the intersection of the lines of sight of the left eye pupil and the right eye pupil into the landing point of the line of sight on the display screen based on the size of the display screen and the distance between the inward-facing camera and the display screen.

[0160] Specifically, the calculation process is as follows:

[0161] Step 1: Define the coordinate system.

[0162] World Space: The global reference system of the user's head or XR glasses.

[0163] Eye Space: The pupil center is the origin and the line of sight is the -Z axis (right-handed system).

[0164] Screen Space: The 2D pixel coordinates of the display screen (the upper left corner is (0,0)).

[0165] Step 2: Get the viewing direction (Ray Casting).

[0166] Assume that the coordinates of the pupil center in the eye coordinate system are (0,0,0) (itself is the origin).

[0167] The sight direction vector is d=(0,0,-1) (looking straight ahead by default).

[0168] Step 3: Calculate the intersection of the line of sight and the display screen.

[0169] Assuming the XR display is a 3D plane (e.g., Z = -d, where d is the distance between the inward-facing camera and the display, i.e., the physical distance from the pupil to the display):

[0170] (1) Parameterized line of sight equation:

[0171] ;

[0172] (2) Find the intersection point between the line of sight and the display screen plane Z = -d:

[0173] ;

[0174] That is, the eye coordinate system coordinates of the line of sight intersection are (dx×d,dy×d,-d).

[0175] Step 4: Convert to screen space.

[0176] (1) Apply the perspective projection matrix: Convert the line of sight intersection (dx×d,dy×d,-d) to clip space through the projection matrix. The calculation formula is as follows:

[0177] ;

[0178] Where P is the perspective projection matrix.

[0179] (2) Perspective division, the calculation formula is as follows:

[0180] ;

[0181] The results are in the NDC (Normalized Device Coordinates) range [-1, 1]².

[0182] (3) NDC → screen pixel coordinates:

[0183] Assuming the screen resolution (W, H), we have the following formula:

[0184] ;

[0185] This embodiment identifies the user's pupil, determines the three-dimensional coordinates of the pupil center in the eye coordinate system, and calculates the line of sight direction vector. Through a preset perspective projection matrix, the line of sight direction is intersected with the display screen plane of the XR glasses to obtain the coordinates of the line of sight intersection in the clipping space. Perspective division is performed on the clipping space coordinates to convert them into normalized device coordinates (NDC). The NDC coordinates are mapped to the pixel coordinate system of the display screen to obtain the 2D position of the line of sight landing point.

[0186] Figure 7 This is a schematic diagram of perspective projection matrix calculation in this embodiment, as shown in Figure 7 As shown, let any point v(x,y,z) in three-dimensional space be the projection of point v on the near surface of the viewing cone. p (x p ,y p ,z p ), the final NDC coordinate is v n (x n ,y n ,z n ). According to the imaging principle, we know that:

[0187] ;

[0188] Where n represents the near clipping plane distance.

[0189] x p and y p to x n and y n The transformation can be regarded as a linear function, because only the near surface is rescaled (linear mapping).

[0190] Taking x as an example, the linear transformation can be written as:

[0191] ;

[0192] Since l corresponds to -1 and r corresponds to 1, we can get:

[0193] ;

[0194] Solving this system of equations yields:

[0195] ;

[0196] Among them, α represents the scaling factor, β represents the translation factor, and r and l represent the left and right boundaries of the near clipping plane respectively.

[0197] The same applies in the y direction.

[0198] Construct the perspective projection matrix. Since the denominator of the isotropic coordinate is z, the fourth row of the perspective projection matrix must be (0 0 1 0). Then, we can get some matrix elements of the transformation matrix as shown below:

[0199] ;

[0200] Where t and b represent the upper and lower boundaries of the near clipping plane, respectively.

[0201] From the symmetry, we know that the z coordinate must be independent of the x and y coordinates, so a and b are both 0, so it can be simplified to:

[0202] ;

[0203] This gives:

[0204] ;

[0205] In order to correspond n to -1, f corresponds to 1, so:

[0206] Among them, c and d represent the depth map parameters, and f represents the far clipping plane distance.

[0207] ;

[0208] Solving for c and d, we finally get the complete perspective projection matrix:

[0209] ;

[0210] Figure 8 A diagram of the text box projection position is provided, such as Figure 8 As shown, a sight line vector (the ray from the eyeball to the display screen) can be determined by the pupil center coordinates to obtain the left eye sight line vector and the right eye sight line vector. The sight line landing point can be determined by calculating the nearest intersection point of the two sight line vectors in 3D space, or the midpoint of the two sight lines or the projection point of the display screen can be taken as the sight line landing point.

[0211] In step S304, the display object is displayed at the sight point of the display screen according to the display angle.

[0212] This embodiment calculates binocular parallax depth through an inward-facing camera to accurately locate the point of sight, and then displays the display object at the point of sight on the display screen according to the display angle. It can still track stably when the user turns his head, simulating the real effect of the user viewing the real object.

[0213] In one embodiment, based on the same inventive concept as the above-mentioned OCR recognition method based on XR glasses, Figure 9 A structural diagram of an OCR recognition device is also provided. Figure 9As shown, the device includes:

[0214] An image acquisition module 21 is used to acquire the current frame image through the outward-facing camera of the XR glasses;

[0215] The first exposure module 22 is configured to extract similar point pairs between the current frame image and the previous frame image, and determine whether the displacement between the similar point pairs is not less than a preset displacement threshold; if it is determined that the displacement between the similar point pairs is not less than the preset displacement threshold, globally expose the current frame image based on the exposure parameters of the previous frame image to obtain a first exposure image;

[0216] The second exposure module 23 is configured to obtain a shooting angle of the outward-facing camera when capturing the current frame image, correct the first exposure image based on the shooting angle, extract a text frame based on the corrected first exposure image, and perform partial exposure of the first exposure image based on the text frame to obtain a second exposure image;

[0217] a recognition module 24 for recognizing characters in a text box in the second exposure image and generating a display object based on the recognized characters;

[0218] The display module 25 is used to restore the display angle of the text box according to the shooting angle, and display the display object on the display screen of the XR glasses according to the display angle.

[0219] In one embodiment, the first exposure module 22 is further configured to perform the following steps:

[0220] Using the previous frame image as a reference object, perform posture correction on the current frame image;

[0221] Extracting similar point pairs between the corrected current frame image and the previous frame image respectively, and calculating the horizontal or vertical displacement of the similar point pairs between the corrected current frame image and the previous frame image;

[0222] Among multiple pairs of similar point pair displacements, remove the maximum displacement and the minimum displacement, calculate the average of the remaining similar point pair displacements, and obtain the target similar point pair displacement;

[0223] Determine whether the displacement of the target similar point pair is not less than the preset displacement threshold.

[0224] In one embodiment, the second exposure module 23 is further configured to perform the following steps:

[0225] The inertial measurement unit in the XR glasses is used to calculate the yaw angle of the inertial measurement unit, and the shooting angle of the outward-facing camera is obtained based on the yaw angle;

[0226] The first exposure image is offset along the Y direction based on the shooting angle to correct the tilt of the first exposure image.

[0227] In one embodiment, the second exposure module 23 is further configured to perform the following steps:

[0228] Taking the pixel area corresponding to the text box as the target pixel area, calculate the brightness histogram of the target pixel area;

[0229] The exposure gain of the target pixel area is increased according to the brightness histogram to obtain a second exposure image.

[0230] In one embodiment, the identification module 24 is further configured to perform the following steps:

[0231] Obtaining the original text of the characters recognized based on the previous frame of image to obtain a first character string;

[0232] Recognize the original characters in the text box to obtain a second character string;

[0233] Comparing the first character string with the second character string to obtain a character string similarity;

[0234] Determine whether the string similarity is not less than a preset similarity threshold;

[0235] If it is determined that the string similarity is not lower than the preset similarity threshold, the translation text of the first string is read from the cache list, and the translation text of the first string is used as the translation text of the second string;

[0236] The translated text of the second character string is displayed.

[0237] In one embodiment, the identification module 24 is further configured to perform the following steps:

[0238] Calculating the edit distance between the first string and the second string;

[0239] Determine string similarity based on edit distance.

[0240] In one embodiment, the display module 25 is further configured to perform the following steps:

[0241] The left and right eye images of the user are captured by the left and right binocular inward-facing cameras of the XR glasses;

[0242] Identify the left pupil in the left eye image and the right pupil in the right eye image;

[0243] Obtaining the size of the display screen and the distance between the inward-facing camera and the display screen, and converting the intersection of the sight lines of the left pupil and the right pupil into a sight line landing point on the display screen based on the size of the display screen and the distance between the inward-facing camera and the display screen;

[0244] The display object is displayed at the viewing point of the display screen according to the display angle.

[0245] In one embodiment, the display module 25 is further configured to perform the following steps:

[0246] Perform threshold segmentation on the left eye image and the right eye image respectively;

[0247] Counting target pixels in the left eye image and the right eye image after threshold segmentation, calculating the pupil area according to the counted number of target pixels, and calculating the pupil radius according to the pupil area;

[0248] Set the window with the upper left corner pixel of the left eye image and the right eye image as the starting point and the radius as the side length;

[0249] Slide the window in the left eye image and the right eye image respectively, and detect the number of black pixels in the window at each sliding;

[0250] The left eye pupil corresponding to the left eye image and the right eye pupil corresponding to the right eye image are determined based on the area that slides to the maximum target pixel number.

[0251] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0252] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described method embodiments. Any reference to a memory, database, or other medium used in the embodiments provided herein may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (XRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), data processing logic devices based on quantum computing, and the like.

[0253] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0254] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. An OCR recognition method based on XR glasses, characterized in that: include: Capture the current frame image through the outward-facing camera of the XR glasses; Extracting similar point pairs between the current frame image and the previous frame image, and determining whether the displacement between the similar point pairs is not less than a preset displacement threshold; When it is determined that the displacement between the pair of similar points is not less than the preset displacement threshold, globally exposing the current frame image based on the exposure parameter of the previous frame image to obtain a first exposure image; obtaining a shooting angle of the outward-facing camera when capturing the current frame image, correcting the first exposure image based on the shooting angle, extracting a text frame based on the corrected first exposure image, and partially exposing the first exposure image based on the text frame to obtain a second exposure image; recognizing characters in the text box in the second exposure image, and generating a display object based on the recognized characters; The display angle of the text box is restored according to the shooting angle, and the display object is displayed on the display screen of the XR glasses according to the display angle.

2. The OCR recognition method based on XR glasses according to claim 1, characterized in that: Extracting similar point pairs between the current frame image and the previous frame image, and determining whether the displacement between the similar point pairs is not less than a preset displacement threshold, including: Taking the previous frame image as a reference object, performing posture correction on the current frame image; respectively extracting similar point pairs between the corrected current frame image and the previous frame image, and calculating the horizontal or vertical displacement of the similar point pairs between the corrected current frame image and the previous frame image; Among multiple pairs of similar point pair displacements, remove the maximum displacement and the minimum displacement, calculate the average of the remaining similar point pair displacements, and obtain the target similar point pair displacement; It is determined whether the displacement of the target similar point pair is not less than the preset displacement threshold.

3. The OCR recognition method based on XR glasses according to claim 1, characterized in that: Acquiring a shooting angle of the outward-facing camera when capturing the current frame image, and correcting the first exposure image based on the shooting angle, including: Calculating the yaw angle of the inertial measurement unit (IMU) in the XR glasses, and obtaining the shooting angle of the outward-facing camera based on the IMU. The first exposure image is offset along the Y direction based on the shooting angle to correct the tilt of the first exposure image.

4. The OCR recognition method based on XR glasses according to claim 1, characterized in that: Partially exposing the first exposure image based on the text box includes: Taking the pixel area corresponding to the text box as the target pixel area, calculating the brightness histogram of the target pixel area; The exposure gain of the target pixel area is increased according to the brightness histogram to obtain the second exposure image.

5. The OCR recognition method based on XR glasses according to claim 1, characterized in that: Recognizing characters in the text box in the second exposure image and generating a display object based on the recognized characters, including: Acquire the original text of the characters recognized based on the previous frame of image to obtain a first character string; Recognize the original characters in the text box to obtain a second character string; Comparing the first character string and the second character string for similarity to obtain a character string similarity; Determining whether the similarity of the character strings is not less than a preset similarity threshold; If it is determined that the string similarity is not lower than the preset similarity threshold, reading the translation text of the first string from the cache list and using the translation text of the first string as the translation text of the second string; The translated text of the second character string is used as the display object.

6. The OCR recognition method based on XR glasses according to claim 5, characterized in that: Comparing the first character string and the second character string for similarity to obtain character string similarity includes: Calculating an edit distance between the first character string and the second character string; The character string similarity is determined according to the edit distance.

7. The OCR recognition method based on XR glasses according to claim 1, characterized in that: Before displaying the display object on the display screen of the XR glasses according to the display angle, the method further includes: The left eye image and the right eye image of the user are captured by the left and right binocular inward-facing cameras of the XR glasses; identifying a left eye pupil in the left eye image and identifying a right eye pupil in the right eye image; Obtaining the size of the display screen and the distance between the inward-facing camera and the display screen, and converting the intersection of the sight lines of the left pupil and the right pupil into a sight line landing point on the display screen based on the size of the display screen and the distance between the inward-facing camera and the display screen; The display object is displayed at the sight point of the display screen according to the display angle.

8. The OCR recognition method based on XR glasses according to claim 7, characterized in that: Identifying a left eye pupil in the left eye image and a right eye pupil in the right eye image, comprising: performing threshold segmentation on the left-eye image and the right-eye image respectively; Counting target pixels in the left-eye image and the right-eye image after threshold segmentation, calculating the pupil area according to the counted number of target pixels, and calculating the pupil radius according to the pupil area; Setting a window with the upper left corner pixel of the left-eye image and the right-eye image as a starting point and the radius as a side length; Sliding the window in the left-eye image and the right-eye image respectively, and detecting the number of black pixels in the window during each sliding; The left eye pupil corresponding to the left eye image and the right eye pupil corresponding to the right eye image are determined according to the area that slides to the maximum target pixel number.

9. An OCR recognition device, characterized in that: include: The image acquisition module is used to capture the current frame image through the outward-facing camera of the XR glasses; a first exposure module configured to extract similar point pairs between the current frame image and the previous frame image, and determine whether a displacement between the similar point pairs is not less than a preset displacement threshold; and when it is determined that the displacement between the similar point pairs is not less than the preset displacement threshold, perform global exposure on the current frame image based on the exposure parameters of the previous frame image to obtain a first exposure image; a second exposure module, configured to obtain a shooting angle of the outward-facing camera when capturing the current frame image, correct the first exposure image based on the shooting angle, extract a text frame based on the corrected first exposure image, and partially expose the first exposure image based on the text frame to obtain a second exposure image; a recognition module, configured to recognize characters in the text box in the second exposure image, and generate a display object based on the recognized characters; The display module is used to restore the display angle of the text box according to the shooting angle, and display the display object on the display screen of the XR glasses according to the display angle.

10. An XR glasses, characterized in that: include: A mirror frame, and an outward-facing camera, a display screen, and a control module installed in the mirror frame; wherein, The outward-facing camera and the display screen are respectively connected to the control module; The control module is configured to execute the steps of the method according to any one of claims 1 to 8.

Images