Image processing method and device, intelligent glasses and storage medium
By collecting and transmitting images on smart glasses to external devices or cloud servers for identification processing, the problem of increased power consumption of smart glasses after the introduction of image recognition algorithm is solved, and power consumption is reduced and battery life is improved.
Patent Information
- Application Number
- CN202311104706.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-29
- Publication Date
- 2025-02-28
AI Technical Summary
After the introduction of image recognition algorithms, the power consumption of smart glasses increases, resulting in a decrease in battery life. How to reduce the power consumption of smart glasses while meeting functional needs has become a challenge.
By calling the camera module on the smart glasses, collecting the original scene image and transmitting it to an electronic device or cloud server connected to the smart glasses, performing image recognition processing, returning the recognition results and displaying them on the smart glasses.
It effectively reduces the power consumption of smart glasses, improves its battery life, and reduces the performance requirements for smart glasses processors, making the device smaller.
Smart Images
Figure CN119540804A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device, smart glasses and computer-readable storage medium. Background Art
[0002] In order to meet the diversified functions of smart glasses, it is necessary to perform algorithmic recognition on images. However, the image recognition algorithm has a large amount of data processing, so the introduction of image recognition algorithms often requires greater power consumption. For smart glasses with limited volume, the power consumption problem has ushered in a huge challenge. Therefore, how to reduce the power consumption of smart glasses to improve the battery life of smart glasses while introducing image recognition algorithms to meet the use functions of smart glasses has become an urgent problem to be solved. Summary of the invention
[0003] The present application provides an image processing method, device, smart glasses and computer-readable storage medium, which can reduce the power consumption of smart glasses and improve the battery life of smart glasses while introducing an image recognition algorithm to meet the usage functions of smart glasses.
[0004] In a first aspect, the present application provides an image processing method using smart glasses, the method comprising:
[0005] When the camera module activation condition of the smart glasses is met, calling the camera module of the smart glasses to collect the original scene image of the target scene area;
[0006] Transmitting the original scene image to an electronic device connected to the smart glasses;
[0007] Acquire an image recognition result fed back by the electronic device based on the original scene image;
[0008] The image recognition result is displayed in a glasses display area of the smart glasses.
[0009] In some embodiments, obtaining the image recognition result fed back by the electronic device based on the original scene image includes:
[0010] If the available resources of the electronic device meet the resources occupied by processing the original scene image, obtaining a first image recognition result obtained by the electronic device processing the original scene image, wherein the first image recognition result is obtained by the electronic device performing recognition based on the original scene image;
[0011] Alternatively, if the available resources of the electronic device do not meet the resources occupied by processing the original scene image, a second image recognition result received by the electronic device from the cloud server is obtained, wherein the second image recognition result is obtained by the cloud server based on the original scene image.
[0012] In some embodiments, obtaining the image recognition result fed back by the electronic device based on the original scene image includes:
[0013] Obtain a third image recognition result received by the electronic device from the cloud server, wherein the third image recognition result is obtained by the cloud server through recognition based on an image preprocessing result of the original scene image, and the image preprocessing result is obtained by the electronic device through processing based on the original scene image.
[0014] In some embodiments, the method further comprises:
[0015] After obtaining the image recognition result fed back by the electronic device based on the original scene image, the camera module of the smart glasses is turned off.
[0016] In some embodiments, the image recognition result includes translated text of the original text in the original scene image, the translated text is obtained by translating the original text in the original scene image, and the original text is obtained by extracting text from the original scene image;
[0017] The displaying of the image recognition result in the glasses display area of the smart glasses includes:
[0018] Obtaining a first position of the original text of the translated text in the original scene image;
[0019] Adding the translated text to the original scene image based on the first position to obtain a translated image of the target scene area;
[0020] The translated image is displayed in a glasses display area of the smart glasses.
[0021] In some embodiments, obtaining the first position of the original text of the translated text in the original scene image includes:
[0022] Acquire a text area of the original text fed back by the electronic device based on the original image, wherein the text area is used to indicate a position of the original text in the original scene image;
[0023] The first position is determined based on the area where the text is located.
[0024] In some embodiments, the image recognition result includes a detection result of a target object in the original scene image;
[0025] The displaying of the image recognition result in the glasses display area of the smart glasses includes:
[0026] Based on the detection result, obtaining a second position of the target object in the original scene image;
[0027] Highlighting the target object in the original scene image based on the second position, obtaining an object recognition image of the target scene area;
[0028] The object recognition image is displayed in a glasses display area of the smart glasses.
[0029] In a second aspect, the present application provides an image processing device, the image processing device comprising:
[0030] A control unit, configured to call the camera module of the smart glasses to collect an original scene image of a target scene area when a condition for turning on the camera module of the smart glasses is met;
[0031] A transmission unit, used to transmit the original scene image to an electronic device connected to the smart glasses;
[0032] The transmission unit is further used to obtain an image recognition result fed back by the electronic device based on the original scene image;
[0033] A display unit is used to display the image recognition result in a glasses display area of the smart glasses.
[0034] In a third aspect, the present application further provides a pair of smart glasses, comprising a main control module and a memory, wherein the memory stores a computer program, and when the main control module calls the computer program in the memory, it executes any image processing method provided in the present application.
[0035] In a fourth aspect, the present application also provides a computer-readable storage medium on which a computer program is stored, and the computer program is loaded by a processor to execute the image processing method.
[0036] In the present application, on the one hand, when the conditions for starting the camera module are met, the camera module of the smart glasses is called to collect the original scene image of the target scene area, so that the camera module can be kept closed when the smart glasses do not need to collect images, reducing the power consumption of the camera module, thereby improving the battery life of the smart glasses; on the second hand, by transmitting the original scene image to an electronic device and obtaining an image recognition result based on the feedback of the electronic device based on the original scene image, and displaying the image recognition result based on the feedback of the electronic device based on the original scene image in the glasses display area of the smart glasses, the image recognition algorithm can be separated from the smart glasses for processing, which can reduce the power consumption of the smart glasses; on the other hand, the processing capability requirements of the smart glasses can be reduced, thereby reducing the performance requirements of the processor chip (i.e., the main control module) of the smart glasses, and thus making the smart glasses more compact; on the third hand, by separating the image recognition algorithm from the smart glasses for processing, since a larger and more accurate image recognition algorithm can be used for processing when separated from the smart glasses, the requirements for image resolution can be reduced to a certain extent, and a low-resolution camera module can be used, thereby reducing the performance requirements of the camera module of the smart glasses, and effectively reducing the module size of the camera module. It can be seen that the embodiments of the present application can reduce the power consumption of smart glasses and improve the battery life of smart glasses while introducing an image recognition algorithm to meet the usage functions of smart glasses. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0038] Figure 1 is a schematic diagram of an embodiment of an image processing system provided by an embodiment of the present application;
[0039] Figure 2 is a schematic block diagram of the structure of a pair of smart glasses provided in an embodiment of the present application;
[0040] Figure 3 It is a flowchart of an image processing method provided by an embodiment of the present application;
[0041] Figure 4 is a schematic diagram for explaining the image processing process in an embodiment of the present application;
[0042] Figure 5 It is a schematic diagram of the structure of an embodiment of the image processing device provided in the embodiments of the present application. DETAILED DESCRIPTION
[0043] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.
[0044] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.
[0045] In the description of the embodiments of the present application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.
[0046] In order to enable any person skilled in the art to implement and use the present application, the following description is provided. In the following description, details are listed for the purpose of explanation. It should be understood that those of ordinary skill in the art can recognize that the present application can also be implemented without using these specific details. In other examples, the known process will not be elaborated in detail to avoid unnecessary details that make the description of the present application embodiment obscure. Therefore, the present application is not intended to be limited to the embodiments shown, but is consistent with the widest range of principles and features disclosed in accordance with the embodiments of the present application.
[0047] The embodiments of the present application provide an image processing method, device, electronic device and computer-readable storage medium, wherein the image processing device can be integrated in an electronic device.
[0048] The executor of the image processing method in the embodiment of the present application may be the image processing device provided in the embodiment of the present application, or an electronic device integrating the image processing device, wherein the image processing device may be implemented in hardware or software.
[0049] Some embodiments of the present application are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.
[0050] Please refer to Figure 1 , Figure 1is a schematic diagram of an embodiment of an image processing system provided by an embodiment of the present application. The image processing method provided by an embodiment of the present application can be applied to Figure 1 In the image processing system shown in FIG. Figure 1 As shown, the image processing system may include smart glasses, electronic devices and cloud servers. Among them, the smart glasses include a main control module, a communication module (such as a Bluetooth module, a WIFI module), and a camera module. The camera module can be connected to the smart glasses through a USB interface, so that after the camera module captures the image, the main control module of the smart glasses obtains the image captured by the camera module through the USB interface; or, the camera module can also be fixed on the smart glasses. Among them, the smart glasses can establish a communication connection with the electronic device through Bluetooth or WIFI to realize data transmission between the smart glasses and the electronic device; the cloud server and the electronic device can be connected through the network to realize data transmission between the cloud server and the electronic device.
[0051] Understandably, Figure 1 This is only an illustration of an embodiment of the image processing system and does not limit the image processing system. For example, in another embodiment of the image processing system, the camera module may also be integrated into the smart glasses as a part of the smart glasses.
[0052] like Figure 2 As shown, Figure 2 It is a schematic block diagram of the structure of a smart glasses provided in an embodiment of the present application. The smart glasses 200 include a main control module 201, a memory 202 and a communication module 203. The main control module 201 and the memory 202 are connected via a bus 204, which is, for example, an I2C (Inter-integrated Circuit) bus.
[0053] Specifically, the main control module 201 is used to provide computing and control capabilities to support the operation of the entire smart glasses 200. The main control module 201 can be a central processing unit (CPU), and the main control module 201 can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.
[0054] Specifically, the memory 202 may be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a mobile hard disk.
[0055] Specifically, the communication module 203 is used to implement data transmission with a peripheral device or a server. The communication module 203 may specifically include but is not limited to a Bluetooth module and a WIFI module.
[0056] Those skilled in the art will understand that Figure 2 The structure shown in the figure is merely a block diagram of a partial structure related to the embodiment scheme of the present application, and does not constitute a limitation on the smart glasses to which the embodiment scheme of the present application is applied. The specific smart glasses may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0057] The main control module 201 is used to run the computer program stored in the memory 202, and implement any one of the image processing methods provided in the embodiments of the present application when executing the computer program. For example, the main control module 201 is used to run the computer program stored in the memory 202, and can implement the following steps when executing the computer program:
[0058] When the conditions for turning on the camera module of the smart glasses are met, the camera module of the smart glasses is called to collect the original scene image of the target scene area; the original scene image is transmitted to an electronic device connected to the smart glasses; an image recognition result based on the feedback of the electronic device based on the original scene image is obtained; and the image recognition result is displayed in the glasses display area of the smart glasses.
[0059] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the smart glasses described above can refer to the corresponding process in the following image processing method embodiment, and will not be repeated here.
[0060] In the following, the image processing method is applied to Figure 1 Taking the image processing system shown in FIG. 1 as an example, the image processing method provided by the embodiment of the present application is described in detail. It should be noted that: Figure 1 and Figure 2 The scenario is only used to explain the image processing method provided in the embodiment of the present application, but does not constitute a limitation on the application scenario of the image processing method provided in the embodiment of the present application.
[0061] See also Figure 3 , Figure 3 301 to 304, wherein:
[0062] 301. When a condition for starting the camera module of the smart glasses is met, the camera module of the smart glasses is called to collect an original scene image of a target scene area.
[0063] The image processing method of the embodiment of the present application can be applied to various business scenarios, for example, it can be used in text translation scenarios, target object (such as face) recognition scenarios, etc.
[0064] Among them, smart glasses can be AR (augmented reality) glasses, VR (Virtual Reality) glasses, etc.
[0065] Among them, the camera module startup condition can be set according to the actual business scenario requirements. For example, it can be set to automatically trigger the command to turn on the camera module of the smart glasses in certain specific scenarios (such as when the user chooses to use the translation function in the smart glasses). At this time, the camera module startup condition can be set to: triggering the translation function; for another example, the camera module of the smart glasses can be automatically triggered to turn on when the inertial sensor (IMU Inertial Measurement Unit, IMU) data (such as IMU data collected within a certain period of time) meets certain conditions. At this time, the camera module startup condition can be set to: detecting that the IMU data collected within a certain period of time meets certain conditions; for another example, the camera module startup condition can be set to receiving a specific sound.
[0066] Among them, the type of camera module can be set according to the actual business scenario requirements. For example, in the scenario where the camera module of the smart glasses is used to collect images for spectrum recognition, the camera module can be a spectral camera, in the scenario where the camera module of the smart glasses is used to collect images for 3D information recognition, the camera module can be a depth camera, and in the scenario where the camera module of the smart glasses is used to collect images for environmental thermal radiation temperature recognition, the camera module can be an infrared camera. In terms of structural setting, the camera module can be integrally formed with the smart glasses, and the camera module can also be connected to the smart glasses through a USB interface. In order to increase the speed at which the main control module of the smart glasses reads the image collected by the camera module, in this embodiment, on the one hand, the camera module is connected to the smart glasses through a USB interface (that is, the camera module is a USB interface camera) to increase the IIC register configuration rate of the camera module, and on the other hand, the smart glasses use a lightweight RTOS (Rral Time Operating System) to minimize the number of programs loaded at startup, thereby increasing the boot speed of the camera module; in this way, the camera module is connected to the smart glasses through a USB interface, and the smart glasses use a lightweight RTOS operating system to achieve a boot speed of the camera module within 1 second.
[0067] The target scene area refers to the scene area to be photographed. For example, the target scene area in the scene of collecting images for text translation is the text area.
[0068] Taking the scene of collecting images for text translation as an example, assuming that the smart glasses trigger the translation function to meet the camera module activation condition, when the smart glasses trigger the translation function, the main control module of the smart glasses can be used to control the camera module of the smart glasses to be activated, and then the camera module of the smart glasses collects the image of the target scene area as the original scene image, so as to translate the text of the target scene area based on the original scene image.
[0069] Taking the scene of collecting images for target object (such as face) recognition as an example, assuming that the face recognition function of smart glasses is detected to meet the camera module activation condition, when the face recognition function of smart glasses is detected to be triggered, the camera module of smart glasses can be controlled to be activated through the main control module of smart glasses, and then the camera module of smart glasses collects the image of the target scene area as the original scene image, so as to identify the face in the target scene area based on the original scene image.
[0070] 302. Transmit the original scene image to an electronic device connected to the smart glasses.
[0071] Among them, electronic devices can be mobile phones, computers, etc.
[0072] Among them, the smart glasses and electronic devices can establish connections through Bluetooth, WIFI, etc. to achieve data interaction between the smart glasses and electronic devices.
[0073] For example, in some embodiments, the smart glasses establish a connection with the electronic device via Bluetooth, and the main control module of the smart glasses can transmit the original scene image to the electronic device connected to the smart glasses via Bluetooth.
[0074] For another example, in some embodiments, the smart glasses establish a connection with the electronic device via WIFI, and the main control module of the smart glasses can transmit the original scene image to the electronic device connected to the smart glasses via WIFI.
[0075] 303. Obtain an image recognition result fed back by the electronic device based on the original scene image.
[0076] There are multiple ways to "obtain the image recognition result fed back by the electronic device based on the original scene image" in step 303, exemplarily including:
[0077] (1) Determine whether to deploy all image recognition processing in the electronic device based on the currently available resources of the electronic device.
[0078] At this time, if the available resources of the electronic device meet the resources occupied by processing the original scene image, the electronic device, after receiving the original scene image sent by the smart glasses, identifies the original scene image to obtain a first image recognition result, and sends the obtained first image recognition result to the smart glasses; thereby, the smart glasses can receive the first image recognition result fed back by the electronic device based on the original scene image; wherein, the first image recognition result is obtained by the electronic device through recognition based on the original scene image. For example, taking the scenario of collecting images for text translation as an example, at this time, the first image recognition result is the translated text of the original text in the original scene image. After the electronic device receives the original scene image sent by the smart glasses, the electronic device calls a preset optical character recognition (OCR) algorithm, and performs text extraction processing on the original scene image through the preset OCR algorithm to obtain the original text in the original scene image and the text area where the original text is located in the original scene image; then, the electronic device translates the original text in the original scene image to obtain the translated text of the original text in the original scene image; the electronic device then sends the translated text of the original text in the original scene image and the text area where the original text is located to the smart glasses, so that the text in the original scene image can be translated without the smart glasses, satisfying the text translation function of the smart glasses while reducing the performance requirements of the main control module of the smart glasses, making the smart glasses more compact and having a stronger battery life. For another example, taking the scenario of collecting images for target object (such as face) recognition as an example, at this time, the first image recognition result is the detection result of the target object in the original scene image. After the electronic device receives the original scene image sent by the smart glasses, the electronic device calls the trained target detection model, and performs target detection processing on the original scene image through the trained target detection model to obtain the detection result of the target object (such as face) in the original scene image; then, the electronic device sends the detection result of the target object in the original scene image to the smart glasses, so that the target object in the original scene image can be detected without the smart glasses, which satisfies the target object detection function of the smart glasses while reducing the performance requirements of the main control module of the smart glasses, making the smart glasses more compact and with stronger battery life.
[0079] If the available resources of the electronic device do not meet the resources occupied by processing the original scene image, then after receiving the original scene image sent by the smart glasses, the electronic device can send the original scene image to the cloud server, and the cloud server will recognize the original scene image to obtain a second image recognition result; the cloud server will then send the second image recognition result obtained based on the recognition of the original scene image to the electronic device, and the electronic device will send the second image recognition result sent by the cloud server to the smart glasses after receiving it, so that the smart glasses can receive the second image recognition result fed back by the electronic device based on the original scene image, wherein the second image recognition result is obtained by recognition by the cloud server based on the original scene image. For example, taking the scenario of collecting images for text translation as an example, at this time, the first image recognition result is the translated text of the original text in the original scene image. After the cloud server receives the original scene image sent by the electronic device, the cloud server calls the preset OCR algorithm, and performs text extraction processing on the original scene image through the preset OCR algorithm to obtain the original text in the original scene image and the text area of the original text in the original scene image; then, the cloud server translates the original text in the original scene image to obtain the translated text of the original text in the original scene image; the cloud server then sends the translated text of the original text in the original scene image and the text area of the original text to the electronic device, which receives it and then feeds it back to the smart glasses, so that the text in the original scene image can be translated without the smart glasses, satisfying the text translation function of the smart glasses while reducing the performance requirements of the main control module of the smart glasses, making the smart glasses more compact and with stronger battery life. For another example, taking the scenario of collecting images for target object (such as face) recognition as an example, at this time, the first image recognition result is the detection result of the target object in the original scene image. After the cloud server receives the original scene image sent by the electronic device, the cloud server calls the trained target detection model, and performs target detection processing on the original scene image through the trained target detection model to obtain the detection result of the target object (such as face) in the original scene image; then, the cloud server sends the detection result of the target object in the original scene image to the electronic device, which receives it and then feeds it back to the smart glasses, so that the target object in the original scene image can be detected without the smart glasses, thereby meeting the target object detection function of the smart glasses while reducing the performance requirements of the main control module of the smart glasses, making the smart glasses more compact and with stronger battery life.
[0080] (2) According to pre-established task processing rules (such as pre-determining which processing operations are to be performed by the electronic device and which are to be performed by the cloud server), determine whether to use the electronic device to perform recognition processing on the original scene image to obtain the image recognition result, or to use the cloud server to perform recognition processing on the original scene image to obtain the image recognition result.
[0081] At this time, if the processing operation corresponding to the processing task of the original scene image is the same as the preset processing operation of the electronic device (such as the preset processing operation of the electronic device includes: spectrum recognition, 3D information recognition, and ambient thermal radiation temperature recognition), then after receiving the original scene image sent by the smart glasses, the electronic device processes the original scene image to obtain a first image recognition result, and sends the obtained first image recognition result to the smart glasses, so that the smart glasses can receive the first image recognition result fed back by the electronic device based on the original scene image; wherein, the first image recognition result is obtained by the electronic device through recognition based on the original scene image.
[0082] If the processing operation corresponding to the processing task of the original scene image is the same as the preset processing operation of the cloud server (such as the preset processing operations of the cloud server include: face recognition, text translation), then after receiving the original scene image sent by the smart glasses, the electronic device sends the original scene image to the cloud server, and the cloud server recognizes the original scene image to obtain a second image recognition result; the cloud server then sends the second image recognition result obtained based on the original scene image recognition to the electronic device, and the electronic device receives the second image recognition result sent by the cloud server and then sends it to the smart glasses, so that the smart glasses can receive the second image recognition result fed back by the electronic device based on the original scene image; wherein, the second image recognition result is obtained by recognition by the cloud server based on the original scene image.
[0083] (3) According to the pre-established task processing rules (such as pre-determining which processing operations are to be performed by the electronic device and which processing operations are to be performed by the cloud server), the original scene image is jointly identified and processed by the electronic device and the cloud server to obtain an image recognition result. At this time, after receiving the original scene image sent by the smart glasses, the electronic device first pre-processes the original scene image (such as image segmentation, image enhancement, etc.) to obtain an image pre-processing result of the original scene image, and then sends the image pre-processing result to the cloud server, which further processes the image pre-processing result to obtain a third image recognition result; the cloud server then sends the third image recognition result to the electronic device, and the electronic device receives the third image recognition result sent by the cloud server and then sends it to the smart glasses, so that the smart glasses can receive the third image recognition result fed back by the electronic device based on the original scene image. The third image recognition result is obtained by the cloud server identifying the image pre-processing result of the original scene image, and the image pre-processing result is obtained by the electronic device processing the original scene image. For example, taking the scenario of collecting images for text translation as an example, at this time, the first image recognition result is the translated text of the original text in the original scene image. First, after the electronic device receives the original scene image sent by the smart glasses, the electronic device performs text area segmentation on the original scene image to obtain a text area map; then, the electronic device sends the segmented text area map as an image preprocessing result to the cloud server. After receiving the image preprocessing result (i.e., the text area map), the cloud server calls a preset OCR algorithm, and performs text extraction processing on the original scene image through the preset OCR algorithm to obtain the original text in the original scene image. The cloud server translates the original text in the original scene image to obtain the translated text of the original text in the original scene map; then, the cloud server sends the translated text of the original text in the original scene map to the electronic device, and the electronic device then sends the translated text of the original text in the original scene image to the smart glasses, so that the text in the original scene image can be translated without the smart glasses, satisfying the text translation function of the smart glasses while reducing the performance requirements of the main control module of the smart glasses, making the smart glasses more compact and with stronger battery life.For another example, taking the scenario of collecting images for target object (such as face) recognition as an example, at this time, the first image recognition result is the detection result of the target object in the original scene image. First, after the electronic device receives the original scene image sent by the smart glasses, the electronic device performs image enhancement on the original scene image, and sends the enhanced image as the image preprocessing result to the cloud server; then, after receiving the image preprocessing result (i.e., the enhanced image), the cloud server calls the trained target detection model, and performs target detection processing on the enhanced image through the trained target detection model to obtain the detection result of the target object (such as face) in the original scene image; then, the cloud server sends the detection result of the target object (such as face) in the original scene image to the electronic device, and the electronic device then sends the detection result of the target object (such as face) in the original scene image to the smart glasses, so that the target object in the original scene image can be detected without the smart glasses, satisfying the target object detection function of the smart glasses while reducing the performance requirements of the main control module of the smart glasses, making the smart glasses more compact and with stronger battery life.
[0084] Exemplarily, the processing flow of the OCR algorithm called by the electronic device or cloud server is as follows: 1. Input: Characterize the image and then input it. 2. Binarization: The amount of information contained in the color image is huge and is not suitable for OCR technology. In order to perform OCR-related calculations faster and better, the image is first binarized so that only the foreground information and background information are left. 3. Image denoising. 4. Tilt correction. 5. Layout analysis: Divide the document image into paragraphs and lines. 6. Character cutting: Separate different characters. 7. Character recognition to obtain recognition results. 8. Post-processing: Correct the recognition results according to the relationship of a specific language context. 9. Output: Output the recognized characters in a certain format of text to obtain the original text in the original scene image.
[0085] Among them, the target detection model called by the electronic device or cloud server is obtained through training, and the trained target detection model may include but is not limited to the following models based on the Convolutional Neural Networks (CNN) algorithm: trained Yolov3 model, Yolov5 or Faster R-CNN, etc. Taking the training of the Yolov5 model to obtain the trained Yolov5 model as the trained target detection model as an example, illustratively, the training process of the target detection model can be specifically as follows: obtain training sample data, a training sample data includes a training sample image and a labeled object (such as a face) area block diagram corresponding to the training sample image, and the training sample image is an image containing an object (such as a face); input the training sample image into the Yolov5 model for processing to obtain a predicted object (such as a face) area block diagram; based on a preset loss function (such as a cross entropy loss function), According to the marked object (such as face) area frame diagram and the predicted object (such as face) area frame diagram, the model loss value is calculated; when the model loss value is greater than the preset loss value, the model parameters of the Yolov5 model are updated, and the step of obtaining training sample data is returned until the model loss value is less than or equal to the preset loss value, and the initial Yolov5 model is obtained; the accuracy of the initial Yolov5 model is verified, and when the initial Yolov5 model passes the accuracy verification, the initial Yolov5 model is determined as the pre-trained Yolov5 model. At this time, the trained Yolov5 model can be used as a trained target detection model for detecting the original scene image, and the prediction box of each object (such as face) in the original scene image is obtained as the detection result of the target object (such as face) in the original scene image.
[0086] Furthermore, a communication connection can also be established between the smart glasses and the cloud server, and the smart glasses can directly send the original scene image to the cloud server for the cloud server to perform recognition processing to obtain the image recognition result.
[0087] Furthermore, a communication connection can also be established between the smart glasses and the cloud server. After the cloud server processes and obtains the image recognition result, the cloud server can also directly send the image recognition result to the smart glasses.
[0088] 304. Display the image recognition result in a glasses display area of the smart glasses.
[0089] For example, taking the scenario of collecting images for text translation as an example, at this time, the first image recognition result is the translated text of the original text in the original scene image. After the electronic device receives the original scene image sent by the smart glasses, the electronic device calls the trained text extraction model, and performs text extraction processing on the original scene image through the trained text extraction model to obtain the original text in the original scene image and the area where the original text is located in the original scene image; then, the electronic device translates the original text in the original scene image to obtain the translated text of the original text in the original scene image; the electronic device then sends the translated text of the original text in the original scene image and the area where the original text is located to the smart glasses. After receiving the translated text of the original text in the original scene image and the text area where the original text is located, the smart glasses first determine the first position of the original text of the translated text in the original scene image based on the text area where the original text is located; then add the translated text at the first position in the original scene image (for example, the translated text of the original text can be superimposed on the original text, or the original text can be replaced with the translated text at the original text), thereby forming a translated image of the target scene area; finally, the smart glasses can display the translated image in the glasses display area of the smart glasses, thereby realizing the display of the image recognition result in the glasses display area of the smart glasses.
[0090] The first position is the position of the original text of the translated text in the original scene image.
[0091] The translated image is an image obtained by adding translated text to the original scene image.
[0092] For another example, taking the scene of collecting images for target object (such as face) recognition as an example, at this time, the first image recognition result is the detection result of the target object in the original scene image. After the electronic device receives the original scene image sent by the smart glasses, the electronic device calls the trained target detection model, performs target detection processing on the original scene image through the trained target detection model, and obtains the detection result of the target object (such as face) in the original scene image; then, the electronic device sends the detection result of the target object in the original scene image to the smart glasses. After receiving the detection result of the target object in the original scene image, the smart glasses first determine the second position of the target object in the original scene image based on the detection result; then, the target object is highlighted at the second position in the original scene image, thereby forming an object recognition image of the target scene area; finally, the smart glasses can display the object recognition image in the glasses display area of the smart glasses, thereby realizing the display of the image recognition result in the glasses display area of the smart glasses.
[0093] The second position is the position of the target object in the original scene image.
[0094] Among them, the object recognition image is an image obtained after highlighting the target object in the original scene image.
[0095] Furthermore, in order to effectively reduce the system power consumption of the smart glasses and increase the standby time of the smart glasses, after obtaining the image recognition result fed back by the electronic device based on the original scene image, the camera module of the smart glasses is turned off. In this way, when the camera module is not needed to capture images, the camera module can be automatically turned off, effectively reducing the system power consumption of the smart glasses and increasing the standby time of the smart glasses.
[0096] like Figure 4 In order to better understand the image processing method of the embodiment of the present application, the following takes "the electronic device is a mobile phone connected to the smart glasses, the main control module of the smart glasses is a microcontroller (MCU), and the camera module of the smart glasses is a camera connected via USB" as an example to illustrate the image processing process, as follows:
[0097] 1. When the conditions for turning on the camera module of the smart glasses are met, the MCU of the smart glasses controls the camera to turn on.
[0098] 2. The camera starts to capture and compress images to obtain the original scene image.
[0099] 3. The MCU of the smart glasses obtains the original scene image captured by the camera through the USB interface.
[0100] 4. The MCU of the smart glasses transmits the original scene image to the mobile phone connected to the smart glasses via Bluetooth or WIFI.
[0101] 5. The mobile phone pre-processes the original scene image and sends the image pre-processing result to the cloud server for further processing. The cloud server processes the image recognition result and returns it to the mobile phone.
[0102] 6. The mobile phone transmits the image recognition results to the MCU of the smart glasses.
[0103] 7. After the MCU of the smart glasses receives the image recognition result, the MCU of the smart glasses controls to turn off the camera.
[0104] From the above content, it can be seen that, on the one hand, when the conditions for turning on the camera module are met, the camera module of the smart glasses is called to collect the original scene image of the target scene area, so that the camera module can be kept closed when the smart glasses do not need to collect images, reducing the power consumption of the camera module, thereby improving the battery life of the smart glasses; on the second hand, by transmitting the original scene image to an electronic device and obtaining an image recognition result based on the feedback of the electronic device based on the original scene image, and displaying the image recognition result based on the feedback of the electronic device based on the original scene image in the glasses display area of the smart glasses, the image recognition algorithm can be separated from the smart glasses for processing, which can reduce the power consumption of the smart glasses; on the other hand, the processing capability requirements of the smart glasses can be reduced, thereby reducing the performance requirements of the processor chip (i.e., the main control module) of the smart glasses, and thus making the smart glasses more compact; on the third hand, by separating the image recognition algorithm from the smart glasses for processing, since a larger and more accurate image recognition algorithm can be used for processing when separated from the smart glasses, the requirements for image resolution can be reduced to a certain extent, and a low-resolution camera module can be used, thereby reducing the performance requirements of the camera module of the smart glasses, and effectively reducing the module size of the camera module. It can be seen that the embodiments of the present application can reduce the power consumption of smart glasses and improve the battery life of smart glasses while introducing an image recognition algorithm to meet the usage functions of smart glasses.
[0105] In addition, in order to better implement the image processing method in the embodiment of the present application, based on the image processing method, an image processing device is also provided in the embodiment of the present application, such as Figure 5 FIG. 5 is a schematic diagram of the structure of an embodiment of an image processing device in an embodiment of the present application. The image processing device 500 includes:
[0106] The control unit 501 is used to call the camera module of the smart glasses to collect the original scene image of the target scene area when the camera module startup condition of the smart glasses is met.
[0107] The transmission unit 502 is used to transmit the original scene image to an electronic device connected to the smart glasses.
[0108] The transmission unit 502 is further configured to obtain an image recognition result fed back by the electronic device based on the original scene image.
[0109] The display unit 503 is used to display the image recognition result in the glasses display area of the smart glasses.
[0110] In some embodiments, the transmission unit 502 is specifically used for:
[0111] If the available resources of the electronic device satisfy the resources occupied by processing the original scene image, a first image recognition result obtained by the electronic device processing the original scene image is obtained, wherein the first image recognition result is obtained by the electronic device performing recognition based on the original scene image; or, if the available resources of the electronic device do not satisfy the resources occupied by processing the original scene image, a second image recognition result received by the electronic device from a cloud server is obtained, wherein the second image recognition result is obtained by the cloud server performing recognition based on the original scene image.
[0112] In some embodiments, the transmission unit 502 is specifically used for:
[0113] Obtain a third image recognition result received by the electronic device from the cloud server, wherein the third image recognition result is obtained by the cloud server through recognition based on an image preprocessing result of the original scene image, and the image preprocessing result is obtained by the electronic device through processing based on the original scene image.
[0114] In some embodiments, the control unit 501 is specifically used to:
[0115] After obtaining the image recognition result fed back by the electronic device based on the original scene image, the camera module of the smart glasses is turned off.
[0116] In some embodiments, the image recognition result includes translated text of the original text in the original scene image, the translated text is obtained by translating the original text in the original scene image, and the original text is obtained by extracting text from the original scene image; the display unit 503 is specifically used to:
[0117] The first position of the original text of the translated text in the original scene image is obtained; based on the first position, the translated text is added to the original scene image to obtain a translated image of the target scene area; and the translated image is displayed in the glasses display area of the smart glasses.
[0118] In some embodiments, the display unit 503 is specifically used for:
[0119] The text area of the original text fed back by the electronic device based on the original image is obtained, wherein the text area is used to indicate the position of the original text in the original scene image; and the first position is determined based on the text area.
[0120] In some embodiments, the image recognition result includes a detection result of a target object in the original scene image; the display unit 503 is specifically used to:
[0121] Based on the detection result, a second position of the target object in the original scene image is obtained; based on the second position, the target object is highlighted in the original scene image to obtain an object recognition image of the target scene area; and the object recognition image is displayed in a glasses display area of the smart glasses.
[0122] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous image processing method embodiments, which will not be repeated here.
[0123] A person skilled in the art will appreciate that all or part of the steps in the above-mentioned image processing method may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.
[0124] To this end, an embodiment of the present application provides a computer-readable storage medium, in which multiple computer programs are stored. The computer programs can be loaded by a processor to execute any image processing method provided by the embodiment of the present application.
[0125] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0126] In the above-mentioned embodiments of the image processing device, computer-readable storage medium, and smart glasses, the description of each embodiment has its own emphasis. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process and beneficial effects of the above-described image processing device, computer-readable storage medium, smart glasses and their corresponding units can refer to the description of the image processing method in the above embodiment, and will not be repeated here.
[0127] The above is a detailed introduction to an image processing method, device, smart glasses and computer-readable storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. An image processing method, characterized in that: Using smart glasses, the method includes: When the camera module activation condition of the smart glasses is met, calling the camera module of the smart glasses to collect the original scene image of the target scene area; Transmitting the original scene image to an electronic device connected to the smart glasses; Acquire an image recognition result fed back by the electronic device based on the original scene image; The image recognition result is displayed in a glasses display area of the smart glasses.
2. The image processing method according to claim 1, characterized in that: The obtaining of the image recognition result fed back by the electronic device based on the original scene image includes: If the available resources of the electronic device meet the resources occupied by processing the original scene image, obtaining a first image recognition result obtained by the electronic device processing the original scene image, wherein the first image recognition result is obtained by the electronic device performing recognition based on the original scene image; Alternatively, if the available resources of the electronic device do not meet the resources occupied by processing the original scene image, a second image recognition result received by the electronic device from the cloud server is obtained, wherein the second image recognition result is obtained by the cloud server based on the original scene image.
3. The image processing method according to claim 1, characterized in that: The obtaining of the image recognition result fed back by the electronic device based on the original scene image includes: Obtain a third image recognition result received by the electronic device from the cloud server, wherein the third image recognition result is obtained by the cloud server through recognition based on an image preprocessing result of the original scene image, and the image preprocessing result is obtained by the electronic device through processing based on the original scene image.
4. The image processing method according to claim 1, characterized in that: The method further comprises: After obtaining the image recognition result fed back by the electronic device based on the original scene image, the camera module of the smart glasses is turned off.
5. The image processing method according to claim 1, characterized in that: The image recognition result includes translated text of the original text in the original scene image, wherein the translated text is obtained by translating the original text in the original scene image, and the original text is obtained by extracting text from the original scene image; The displaying of the image recognition result in the glasses display area of the smart glasses includes: Obtaining a first position of the original text of the translated text in the original scene image; Adding the translated text to the original scene image based on the first position to obtain a translated image of the target scene area; The translated image is displayed in a glasses display area of the smart glasses.
6. The image processing method according to claim 5, characterized in that: The obtaining of the first position of the original text of the translated text in the original scene image includes: Acquire a text area of the original text fed back by the electronic device based on the original image, wherein the text area is used to indicate a position of the original text in the original scene image; The first position is determined based on the area where the text is located.
7. The image processing method according to claim 1, characterized in that: The image recognition result includes a detection result of a target object in the original scene image; The displaying of the image recognition result in the glasses display area of the smart glasses includes: Based on the detection result, obtaining a second position of the target object in the original scene image; Highlighting the target object in the original scene image based on the second position, obtaining an object recognition image of the target scene area; The object recognition image is displayed in a glasses display area of the smart glasses.
8. An image processing device, characterized in that: The image processing device comprises: A control unit, configured to call the camera module of the smart glasses to collect an original scene image of a target scene area when a condition for turning on the camera module of the smart glasses is met; A transmission unit, used to transmit the original scene image to an electronic device connected to the smart glasses; The transmission unit is further used to obtain an image recognition result fed back by the electronic device based on the original scene image; A display unit is used to display the image recognition result in a glasses display area of the smart glasses.
9. A smart glasses, characterized in that: The method comprises a main control module and a memory, wherein a computer program is stored in the memory, and when the main control module calls the computer program in the memory, the image processing method according to any one of claims 1 to 7 is executed.
10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is loaded by a processor to execute the image processing method according to any one of claims 1 to 7.