Image processing method, image recognition device, electronic device and medium

By concurrently collecting and stitching images in the image recognition device and using lightweight and general optical character recognition models, the problems of real-time and accuracy of word recognition in the prior art are solved, and the user experience is improved.

CN114663901BActive Publication Date: 2024-11-26BEIJING SOGOU TECHNOLOGY DEVELOPMENT CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202011409362.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-12-04
Publication Date
2024-11-26
Estimated Expiration
2040-12-04

AI Technical Summary

Technical Problem

In order to improve accuracy, existing image recognition devices usually use optical character recognition models when recognizing text, but this will reduce the real-time performance of the intermediate stitching image, resulting in a longer response time on the screen of the first character, affecting the user experience.

Method used

By using the first thread in the image recognition device to collect images in real time and store them in the image stitching module, the second thread splices the images in real time, uses a lightweight optical character recognition model to recognize the intermediate stitching image, and uses a general purpose when the state changes. The optical character recognition model performs final recognition of the image set.

Benefits of technology

It improves the real-time recognition of intermediate stitching images, shortens the response time of the first character on the screen, and ensures the accuracy and user experience of the final recognition results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114663901B_ABST
    Figure CN114663901B_ABST
Patent Text Reader

Abstract

The embodiment of the present invention discloses an image processing method. In the process of real-time image acquisition by an image recognition device, the acquired real-time image is stored in an image stitching module by a first thread; the images in the image stitching module are stitched in real time by a second thread to obtain an intermediate stitching image; the intermediate stitching image is subjected to text recognition by a lightweight optical character recognition model to obtain an intermediate recognition result and display it; and when a state change of the image recognition device is detected, an image set composed of all images in the image stitching module is obtained, and text recognition is performed on the image set by a universal optical character recognition model to obtain a final recognition result and display it. The image processing method, image recognition device, electronic device and medium provided by the embodiment of the present invention can improve the real-time recognition of text recognition for the intermediate stitching image and shorten the response time for the first character to appear on the screen.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of image processing technology, and in particular to an image processing method, an image recognition device, an electronic device, and a medium. Background Art

[0002] With the rapid development of Internet technology, more and more applications are migrating to the Internet. For the convenience of users, users can use image recognition devices such as dictionary pens or reading pens to obtain non-source language texts, and then translate or annotate the obtained texts through Internet data to improve user learning efficiency.

[0003] In the prior art, when using an image recognition device to obtain text, an image corresponding to the text is usually collected, and then text recognition is performed on the image to identify the text or sentence. The computing power of the image recognition device is limited. At this time, in order to improve the recognition accuracy, an optical character recognition model is usually set in the image recognition device to perform text recognition. The higher the recognition accuracy, the worse the real-time performance of the optical character recognition model in recognizing the intermediate spliced ​​image, but the final result will be more accurate. Summary of the invention

[0004] The embodiments of the present invention provide an image processing method, an image recognition device, an electronic device and a medium, which can improve the real-time recognition of text in an intermediate spliced ​​image, ensure the accuracy of the final recognition result, and shorten the response time for the first character to appear on the screen.

[0005] A first aspect of an embodiment of the present invention provides an image processing method, including:

[0006] In the process of collecting images in real time by the image recognition device, the collected real-time images are stored in the image stitching module by the first thread;

[0007] In the process of storing the collected real-time images through the first thread, the images in the image stitching module are stitched in real time through the second thread to obtain an intermediate stitched image;

[0008] Performing text recognition on the intermediate spliced ​​image by using a lightweight optical character recognition model, obtaining an intermediate recognition result and displaying it;

[0009] When a state change of the image recognition device is detected, an image set consisting of all images in the image stitching module is obtained, text recognition is performed on the image set through a universal optical character recognition model, and a final recognition result is obtained and displayed.

[0010] Optionally, the performing real-time stitching of the images in the image stitching module by the second thread to obtain an intermediate stitched image includes:

[0011] The second thread is started through the image stitching module, and the images in the image stitching module are stitched in real time through the second thread to obtain the intermediate stitching image.

[0012] Optionally, performing text recognition on the intermediate spliced ​​image by using a lightweight optical character recognition model to obtain and display an intermediate recognition result includes:

[0013] If the frame number of the intermediate stitched image is not less than the preset frame number, the lightweight optical character recognition model is used to perform text recognition on the intermediate stitched image to obtain and display the intermediate recognition result.

[0014] Optionally, the performing text recognition on the intermediate spliced ​​image by using the lightweight optical character recognition model includes:

[0015] The intermediate spliced ​​image is subjected to text recognition by using the lightweight optical character recognition model, wherein the lightweight optical character recognition model uses a single-core processor in the image recognition device.

[0016] A second aspect of an embodiment of the present invention provides an image recognition device, including:

[0017] An image storage unit, used for storing the collected real-time image in the image stitching module through the first thread during the process of collecting the image in real time through the image recognition device;

[0018] An image stitching unit, configured to stitch the images in the image stitching module in real time through a second thread to obtain an intermediate stitched image during the process of storing the real-time images collected through the first thread;

[0019] An intermediate character recognition unit, used for performing character recognition on the intermediate spliced ​​image by using a lightweight optical character recognition model, obtaining an intermediate recognition result and displaying it;

[0020] The final text recognition unit is used to obtain an image set composed of all images in the image stitching module when a state change of the image recognition device is detected, perform text recognition on the image set through a universal optical character recognition model, obtain a final recognition result and display it.

[0021] Optionally, the image stitching unit is used to start the second thread through the image stitching module, and stitch the images in the image stitching module in real time through the second thread to obtain the intermediate stitched image.

[0022] Optionally, if the number of frames of the intermediate stitched image is not less than a preset number of frames, the intermediate text recognition unit is used to perform text recognition on the intermediate stitched image through the lightweight optical character recognition model, obtain the intermediate recognition result and display it.

[0023] Optionally, the intermediate text recognition unit is used to perform text recognition on the intermediate stitched image through the lightweight optical character recognition model, wherein the lightweight optical character recognition model uses a single-core processor in the image recognition device.

[0024] A third aspect of an embodiment of the present invention provides a device for data processing, comprising a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, wherein the one or more programs include steps for the image processing method as described above.

[0025] A fourth aspect of an embodiment of the present invention provides a machine-readable medium having instructions stored thereon, which, when executed by one or more processors, enables a device to perform the steps of the above-mentioned image processing method.

[0026] The beneficial effects of the embodiments of the present invention are as follows:

[0027] Based on the above technical solution, the real-time collected images are stored in the image stitching module through the first thread. In the process of storing the collected real-time images through the first thread, the images in the image stitching module are stitched in real time through the second thread. In this way, the loss rate of the real-time collected images can be effectively reduced in a concurrent manner. When the image loss rate is reduced, the clarity of the effectively stitched images can be improved, thereby ensuring the accuracy of the final recognition result. The concurrent manner can effectively reduce the stitching delay. Moreover, since lightweight OCR is used to perform text recognition on the intermediate stitched images, the processor load of the image recognition device can be reduced, and the real-time recognition of text recognition on the intermediate stitched images can be improved. When the real-time recognition is improved, the response time for the first character to appear on the screen can be shortened, thereby providing a better user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A flowchart of an image processing method according to an embodiment of the present invention;

[0029] Figure 2 is a schematic diagram of the structure of an image recognition device in an embodiment of the present invention;

[0030] Figure 3 It is a structural block diagram of an image recognition device used as a device in an embodiment of the present invention;

[0031] Figure 4 It is a structural block diagram of the server in some embodiments of the present invention. DETAILED DESCRIPTION

[0032] In order to better understand the above technical scheme, the technical scheme of the embodiment of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiment of the present invention and the specific features in the embodiments are detailed descriptions of the technical scheme of the embodiment of the present invention, rather than limitations on the technical scheme of the present invention. In the absence of conflict, the embodiment of the present invention and the technical features in the embodiments can be combined with each other.

[0033] In response to the technical problem of high delay in intermediate recognition results during text recognition by an image recognition device, an embodiment of the present invention provides an image processing solution, which is used to store the collected real-time images in an image stitching module through a first thread during real-time image acquisition by an image recognition device; in the process of storing the collected real-time images by the first thread, stitch the images in the image stitching module in real time through a second thread to obtain an intermediate stitched image; perform text recognition on the intermediate stitched image through a lightweight optical character recognition model to obtain and display an intermediate recognition result; and when a state change of the image recognition device is detected, obtain an image set consisting of all images in the image stitching module, perform text recognition on the image set through a universal optical character recognition model, obtain a final recognition result, and display it.

[0034] In this way, in the process of storing the collected real-time images through the first thread, the images in the image stitching module are also stitched in real time through the second thread to obtain an intermediate stitched image, and after obtaining the intermediate stitched image, the lightweight OCR model is used to perform text recognition on the intermediate stitched image. In this way, the data processing efficiency is improved in a concurrent manner, and the lightweight OCR model occupies less resources and has high recognition efficiency. In this way, the real-time recognition of text for the intermediate stitched image can be effectively improved. When the real-time recognition is improved, the response time for the first character to appear on the screen can be shortened, thereby providing a better user experience.

[0035] like Figure 1 As shown, an embodiment of the present invention provides an image processing method, comprising the following steps:

[0036] S101, in the process of real-time image acquisition by the image recognition device, storing the acquired real-time image in the image stitching module by the first thread;

[0037] S102, in the process of storing the collected real-time images through the first thread, stitching the images in the image stitching module in real time through the second thread to obtain an intermediate stitching image;

[0038] S103, performing text recognition on the intermediate spliced ​​image by using a lightweight optical character recognition model, obtaining an intermediate recognition result and displaying it; and

[0039] S104, when a state change of the image recognition device is detected, an image set consisting of all images in the image stitching module is obtained, and text recognition is performed on the image set through a general optical character recognition model to obtain and display a final recognition result.

[0040] In the embodiments of this specification, when the image recognition device performs text recognition on an image set through a general optical character recognition model, the recognition data may be provided by a server connected to the image recognition device, and this specification does not impose any specific limitation.

[0041] In step S101, the image recognition device can generally be a device for recognizing text, such as a dictionary pen and a reading pen. At this time, in the process of using the image recognition device to recognize text, the image recognition device will collect images of the target object in real time. The target object is generally an electronic book or a paper book. In this way, in the process of real-time image collection by the image recognition device, the first thread in the image recognition device is started, and the collected real-time image is placed in the image stitching module of the image recognition device through the first thread. The collected real-time images can be arranged in chronological order or randomly arranged. When randomly arranged, the serial number of each frame of the image needs to be stored, and the serial number is set according to the collection time of each frame of the image.

[0042] For example, taking dictionary pen A as an example, when the user uses dictionary pen A to touch and move the target text in book B, the camera of dictionary pen A is turned on, and the camera collects images in real time during the movement of dictionary pen A on the target text. At this time, the first thread in dictionary pen A is started, and the real-time collected images are placed in the image stitching module through the first thread.

[0043] Next, step S102 is executed. In this step, the second thread can be started through the image stitching module or the image recognition device, and this specification does not make any specific limitation.

[0044] And, after starting the second thread and in the process of storing the collected real-time images through the first thread, the images in the image stitching module are stitched in real time through the second thread to obtain an intermediate stitched image.

[0045] Specifically, each time one or more frames of images are placed in the image stitching module, the images placed in the image stitching module are stitched in real time through the second thread to obtain an intermediate stitched image. Since the second thread stitches the images placed in the image stitching module in real time, the intermediate stitched image will change with time.

[0046] For example, taking dictionary pen A as an example, in the process of real-time acquisition of images of the target text by dictionary pen A, the real-time acquired images are stored frame by frame in the image stitching module through the first thread. If at a certain moment, the images placed in the image stitching module are the first frame image represented by C1, the second frame image C2 and the third frame image C3, since the image stitching module starts the second thread, at this time, in the process of storing the acquired real-time images by the first thread, C1, C2 and C3 are stitched together by the second thread to obtain the intermediate stitched image D1; and at a moment after a certain moment, a new frame image is added to the image stitching module. Like C4, at this time, C1, C2, C3 and C4 are stitched together by the second thread to obtain an intermediate stitched image D2; and so on, each time a new frame of image is added to the image stitching module, the second thread stitches the new frame of image to obtain an intermediate stitched image; in this way, the real-time acquired image is stored in the image stitching module by the first thread, which can effectively ensure that the loss rate of the real-time acquired image is reduced; and in the process of storing the acquired real-time image by the first thread, the image in the image stitching module is stitched together in real time by the second thread, which can ensure the real-time nature of the image stitching and the clarity of the intermediate stitched image.

[0047] After the intermediate stitched image is acquired in step S102, step S103 is executed to perform text recognition on the intermediate stitched image in real time using a lightweight optical character recognition model to obtain and display an intermediate recognition result.

[0048] Specifically, since in step S102, in the process of storing the collected real-time images through the first thread, the images in the image stitching module are stitched in real time through the second thread, so that the intermediate stitched image also changes in real time, in this way, when the intermediate stitched image is subjected to text recognition through the lightweight optical character recognition model in step S103, it can be first determined whether the frame number of the intermediate stitched image is not less than the preset frame number, if it is less, no recognition is performed; if it is not less, it can be determined that the length of the text corresponding to the intermediate stitched image is usually greater than the set length, and then the intermediate stitched image can be subjected to text recognition through the lightweight optical character recognition model, and the intermediate recognition result is obtained and displayed; thereafter, the current intermediate stitched image can be subjected to text recognition through the lightweight optical character recognition model at set intervals; and the newly added intermediate stitched image can also be subjected to text recognition through the lightweight optical character recognition model when one or more frames of the intermediate stitched image are added, and the intermediate recognition result is obtained and displayed, at which time, the real-time nature of the intermediate recognition result can be effectively ensured.

[0049] In the embodiments of this specification, the set duration can be set manually or by an image recognition device, or can be set according to the user's usage habits. For example, it can be 30 milliseconds, 40 milliseconds, 50 milliseconds, etc. This specification does not impose any specific restrictions.

[0050] In the embodiments of the present specification, the preset number of frames can collect usage data of a large number of users using the image recognition device, and determine the average number of frames that users use the image recognition device to collect a complete text based on the usage data; determine the preset number of frames based on the average number of frames, at which time, the preset number of frames is usually equal to the average number of frames, and can also be greater than or less than the average number of frames, and can also be set manually or by the image recognition device; and the set length is usually the length of a complete word, and the set length can also be set manually or by the image recognition device, and this specification does not impose any specific restrictions.

[0051] Specifically, when the lightweight optical character recognition (OCR) model detects that the frame number of the intermediate spliced ​​image is not less than the preset frame number, the intermediate spliced ​​image is placed in the lightweight OCR model for text recognition to obtain the intermediate recognition result. Since the preset frame number is usually the average frame number of a complete text, and the frame number of the intermediate spliced ​​image is not less than the preset frame number, it can ensure that the probability that the intermediate recognition result contains the complete text is high; and because the lightweight OCR model used has the effect of small calculation amount and high calculation efficiency, it can effectively improve the real-time performance of the recognized intermediate recognition result. In this way, when the user uses the image recognition device to collect images, the lightweight OCR model can display the intermediate recognition result in real time, so that the user experience is better.

[0052] Specifically, in order to ensure that the intermediate recognition results are displayed in real time, it can be ensured that the lightweight OCR model uses the single-core processor in the image recognition device. The single-core processor can be used exclusively by the lightweight OCR model, that is, the single-core processor is only used by the lightweight OCR model and will not be occupied by other programs, so that the lightweight OCR model can run all the time, and the intermediate spliced ​​image can be entered into the lightweight OCR model for calculation, thereby ensuring the real-time nature of the calculated intermediate recognition results.

[0053] And, when a state change of the image recognition device is detected, step S104 is executed.

[0054] Specifically, when it is detected that the posture of the image recognition device relative to the target object has changed, it can be determined that the state of the image recognition device has changed; specifically, when it is detected that the posture of the image recognition device relative to the target object is adjusted from a contact state to a distant state, it is determined that the state of the image recognition device has changed; or, when it is detected that the state of the image acquisition device of the image recognition device is adjusted from an on state to a off state, it can be determined that the state of the image recognition device has changed; or, when it is detected that the posture of the image recognition device relative to the target object is adjusted from a contact state to a distant state and the duration of the image recognition device in the distant state is greater than a preset time, it is determined that the state of the image recognition device has changed. At this time, the accuracy of judging whether the user operates the image recognition device can be higher, the probability of misrecognition can be reduced, and the user experience can be better.

[0055] Specifically, when the state change of the image recognition device is detected, an image set consisting of all images in the image stitching module is obtained, and then the general OCR model is used to perform text recognition on the image set to obtain and display the final recognition result. At this time, the general OCR model is larger and has higher recognition accuracy than the lightweight OCR model.

[0056] For example, taking dictionary pen A as an example, in the process of real-time image acquisition for the target text through dictionary pen A, the real-time acquired images are stored frame by frame in the image stitching module through the first thread. If at a certain moment, and in the process of storing the acquired real-time images through the first thread, the images in the image stitching module are stitched in real time through the second thread, it is detected whether the frame number of the real-time intermediate stitching image is not less than the preset frame number; if not less than, the intermediate stitching image not less than the preset frame number is put into the lightweight OCR model, and the calculation is performed through the first single-core CPU exclusively occupied by the lightweight OCR model to obtain the intermediate stitching result. If the intermediate stitching result is "I love", "I" is displayed; subsequently, the new intermediate stitching image is input into the lightweight OCR model every 40 milliseconds for text recognition, and the intermediate recognition results are "I love", "I love China" and "I love China" respectively, and the recognized intermediate recognition results are displayed.

[0057] In addition, posture detection is performed on the dictionary pen A. If it is detected through posture detection that the relative posture of the dictionary pen A and the target text changes from a contact state to a distance state and the duration is greater than 20 milliseconds, it is determined that the state of the dictionary pen A has changed, and then an image set consisting of all images in the image stitching module is obtained; then a general OCR model is used to perform text recognition on the image set, and the final recognition result is obtained and displayed, and the final recognition result displayed is "I love China".

[0058] In this way, in the process of storing the collected real-time images through the first thread, the images in the image stitching module are also stitched in real time through the second thread. In this way, the loss rate of the collected images can be effectively reduced in a concurrent manner. When the image loss rate is reduced, the clarity of the stitched images can be effectively improved, thereby ensuring the accuracy of the final recognition result. The concurrent method can effectively reduce the stitching delay. Moreover, since lightweight OCR is used to perform text recognition on the intermediate stitched images, the processor load of the image recognition device can be reduced, and the real-time recognition of text recognition on the intermediate stitched images can be improved. When the real-time recognition is improved, the response time for the first character to appear on the screen can be shortened, thereby providing a better user experience.

[0059] Device Embodiment

[0060] Reference Figure 2 , shows a structural block diagram of an embodiment of an image recognition device of the present invention, which may specifically include:

[0061] The image storage unit 201 is used to store the collected real-time image in the image stitching module through the first thread during the process of collecting the image in real time through the image recognition device;

[0062] An image stitching unit 202 is used to stitch the images in the image stitching module in real time through a second thread to obtain an intermediate stitched image during the process of storing the collected real-time images through the first thread;

[0063] An intermediate character recognition unit 203 is used to perform character recognition on the intermediate spliced ​​image by using a lightweight optical character recognition model, obtain an intermediate recognition result and display it;

[0064] The final text recognition unit 204 is used to obtain an image set composed of all images in the image stitching module when a state change of the image recognition device is detected, perform text recognition on the image set through a universal optical character recognition model, obtain a final recognition result and display it.

[0065] In an optional implementation, the image stitching unit 202 is used to start the second thread through the image stitching module, and stitch the images in the image stitching module in real time through the second thread to obtain the intermediate stitched image.

[0066] In an optional implementation, if the frame number of the intermediate stitched image is not less than a preset frame number, the intermediate text recognition unit 203 is used to perform text recognition on the intermediate stitched image through the lightweight optical character recognition model, obtain the intermediate recognition result and display it.

[0067] In an optional implementation, the intermediate text recognition unit 203 is used to perform text recognition on the intermediate spliced ​​image by using the lightweight optical character recognition model, wherein the lightweight optical character recognition model uses a single-core processor in the image recognition device.

[0068] In an optional implementation, the image recognition device includes a dictionary pen and a reading pen.

[0069] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0070] The various embodiments of the present invention are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to in detail.

[0071] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0072] Figure 39 is a structural block diagram of an image recognition device as a device according to an exemplary embodiment. For example, the device 900 can be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0073] Reference Figure 3 , the device 900 may include one or more of the following components: a processing component 902 , a memory 904 , a power component 906 , a multimedia component 908 , an audio component 910 , an input / output (I / O) interface 912 , a sensor component 914 , and a communication component 916 .

[0074] The processing component 902 generally controls the overall operation of the device 900, such as operations associated with display, incoming calls, data communications, camera operations, and recording operations. The processing component 902 may include one or more processors 920 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 902 may include one or more modules to facilitate the interaction between the processing component 902 and other components. For example, the processing component 902 may include a multimedia module to facilitate the interaction between the multimedia component 908 and the processing component 902.

[0075] The memory 904 is configured to store various types of data to support operations on the device 900. Examples of such data include instructions for any application or method operating on the device 900, contact data, caller book data, messages, pictures, videos, etc. The memory 904 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0076] The power supply component 906 provides power to the various components of the device 900. The power supply component 906 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 900.

[0077] The multimedia component 908 includes a screen that provides an output interface between the device 900 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide motion action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 908 includes a front camera and / or a rear camera. When the device 900 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0078] The audio component 910 is configured to output and / or input audio signals. For example, the audio component 910 includes a microphone (MIC), and when the device 900 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 904 or sent via the communication component 916. In some embodiments, the audio component 910 also includes a speaker for outputting audio signals.

[0079] I / O interface 912 provides an interface between processing component 902 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0080] The sensor assembly 914 includes one or more sensors for providing various aspects of status assessment for the device 900. For example, the sensor assembly 914 can detect the open / closed state of the device 900, the relative positioning of components, such as the display and keypad of the device 900, and the sensor assembly 914 can also detect the position change of the device 900 or a component of the device 900, the presence or absence of user contact with the device 900, the orientation or acceleration / deceleration of the device 900, and the temperature change of the device 900. The sensor assembly 914 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 914 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 914 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0081] The communication component 916 is configured to facilitate wired or wireless communication between the device 900 and other devices. The device 900 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 916 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 916 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0082] In an exemplary embodiment, the apparatus 900 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above method.

[0083] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 904 including instructions, and the instructions can be executed by the processor 920 of the device 900 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0084] Figure 4 1 is a block diagram of the structure of the server in some embodiments of the present invention. The server 1900 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1922 (for example, one or more processors) and memory 1932, and one or more storage media 1930 (for example, one or more mass storage devices) storing application programs 1942 or data 1944. Among them, the memory 1932 and the storage medium 1930 can be short-term storage or permanent storage. The program stored in the storage medium 1930 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the server. Furthermore, the central processing unit 1922 can be configured to communicate with the storage medium 1930 and execute a series of instruction operations in the storage medium 1930 on the server 1900.

[0085] The server 1900 may also include one or more power supplies 1926, one or more wired or wireless network interfaces 1950, one or more input and output interfaces 1958, one or more keyboards 1956, and / or, one or more operating systems 1941, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0086] A non-temporary computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a device (equipment or server), enables the device to execute an image processing method, in the process of real-time image acquisition by an image recognition device, storing the acquired real-time image in an image stitching module by a first thread; in the process of storing the acquired real-time image by the first thread, stitching the images in the image stitching module in real time by a second thread to obtain an intermediate stitched image; performing text recognition on the intermediate stitched image by a lightweight optical character recognition model, obtaining an intermediate recognition result and displaying it; and when a state change of the image recognition device is detected, obtaining an image set composed of all images in the image stitching module, performing text recognition on the image set by a universal optical character recognition model, obtaining a final recognition result and displaying it.

[0087] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that has the functions specified in one or more boxes.

[0088] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0089] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0090] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0091] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. An image processing method, applied to a dictionary pen and a point reading pen, characterized in that: include: In the process of collecting images in real time by the image recognition device, the collected real-time images are stored in the image stitching module by the first thread; In the process of storing the collected real-time images through the first thread, the images in the image stitching module are stitched in real time through the second thread to obtain an intermediate stitched image; Performing text recognition on the intermediate spliced ​​image by using a lightweight optical character recognition model, obtaining an intermediate recognition result and displaying it; and when a state change of the image recognition device is detected, an image set composed of all images in the image stitching module is obtained, text recognition is performed on the image set through a universal optical character recognition model, and a final recognition result is obtained and displayed; the state change specifically includes: when it is detected that the posture of the image recognition device relative to the target object is adjusted from a contact state to a distance state, it is determined that the state of the image recognition device has changed; Alternatively, when it is detected that the state of the image acquisition device of the image recognition device is adjusted from an on state to a off state, it can be determined that the state of the image recognition device has changed; or, when it is detected that the posture of the image recognition device relative to the target object is adjusted from a contact state to a distant state and the duration of the image recognition device in the distant state is greater than a preset time, it is determined that the state of the image recognition device has changed.

2. The method according to claim 1, characterized in that The step of performing real-time stitching of the images in the image stitching module by the second thread to obtain an intermediate stitched image includes: The second thread is started through the image stitching module, and the images in the image stitching module are stitched in real time through the second thread to obtain the intermediate stitching image.

3. The method according to claim 2, characterized in that The step of performing text recognition on the intermediate spliced ​​image by using a lightweight optical character recognition model, obtaining and displaying an intermediate recognition result, includes: If the frame number of the intermediate stitched image is not less than the preset frame number, the lightweight optical character recognition model is used to perform text recognition on the intermediate stitched image to obtain and display the intermediate recognition result.

4. The method according to claim 3, characterized in that The performing text recognition on the intermediate spliced ​​image by using the lightweight optical character recognition model includes: The intermediate spliced ​​image is subjected to text recognition by using the lightweight optical character recognition model, wherein the lightweight optical character recognition model uses a single-core processor in the image recognition device.

5. An image recognition device, applied to a dictionary pen and a point reading pen, characterized in that: include: An image storage unit, used to store the collected real-time image in the image stitching module through the first thread during the process of collecting the image in real time through the image recognition device; An image stitching unit, configured to stitch the images in the image stitching module in real time through a second thread to obtain an intermediate stitched image during the process of storing the real-time images collected through the first thread; An intermediate character recognition unit, used for performing character recognition on the intermediate spliced ​​image through a lightweight optical character recognition model, obtaining an intermediate recognition result and displaying it; The final text recognition unit is used to obtain an image set composed of all images in the image stitching module when detecting a state change of the image recognition device, perform text recognition on the image set through a universal optical character recognition model, obtain a final recognition result and display it; the state change specifically includes: when detecting that the posture of the image recognition device relative to the target object is adjusted from a contact state to a distance state, determining that the state of the image recognition device has changed; Alternatively, when it is detected that the state of the image acquisition device of the image recognition device is adjusted from an on state to a off state, it can be determined that the state of the image recognition device has changed; or, when it is detected that the posture of the image recognition device relative to the target object is adjusted from a contact state to a distant state and the duration of the image recognition device in the distant state is greater than a preset time, it is determined that the state of the image recognition device has changed.

6. The device according to claim 5, characterized in that The image stitching unit is used to start the second thread through the image stitching module, and stitch the images in the image stitching module in real time through the second thread to obtain the intermediate stitching image.

7. The device according to claim 6, characterized in that The intermediate text recognition unit is used to perform text recognition on the intermediate spliced ​​image through the lightweight optical character recognition model if the frame number of the intermediate spliced ​​image is not less than the preset frame number, obtain the intermediate recognition result and display it.

8. The device according to claim 7, characterized in that The intermediate text recognition unit is used to perform text recognition on the intermediate spliced ​​image through the lightweight optical character recognition model, wherein the lightweight optical character recognition model uses a single-core processor in the image recognition device.

9. A device for data processing, characterized in that: The device comprises a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, and the one or more programs include the method steps according to any one of claims 1 to 4.

10. A machine-readable medium having instructions stored thereon, which, when executed by one or more processors, cause an apparatus to perform the method of one or more of claims 1-4.

Citation Information

Patent Citations

  • Method and system for inputting characters

    CN102063611A

  • Model generation method, text recognition method and device, equipment and storage medium

    CN110766050A

  • Method for processing image and electronic device supporting the same

    EP3301905A1

  • Method for outputting consecutive characters in video-recording mode

    US20100272360A1

  • Systems and methods for detecting and classifying objects in video captured using mobile devices

    US20140327940A1