Control method and device based on image-text identification recognition

Abstract

The invention discloses a control method and device based on image-text identification recognition. The method comprises the following steps: acquiring a scene image of a current scene; inputting the scene image into the YOLO model to obtain a local area image of the identification object in the scene image and coordinates of the identification object in the scene image; the local area image is input into a MobileFaceNet model to obtain an identification result of image-text identification content of the identification object, and the image-text identification content comprises at least one of characters or graphic identification; and executing operation in the current scene based on the identification result of the image-text identification content of the identification object and the coordinate of the identification object in the scene image.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a control method and device based on graphic and text identification. Background Art

[0002] With the development of various mobile devices (such as intelligent robots), there is a need for them to recognize graphic symbols, such as elevator buttons. However, due to limitations such as low image quality and the incompatibility between model algorithm size and hardware cost, current mobile devices struggle to achieve stable and accurate recognition of graphic symbols. Summary of the Invention

[0003] According to an embodiment of the present invention, a control method based on graphic identification recognition includes: obtaining a scene image of the current scene; inputting the scene image into a YOLO model to obtain a local area image of an identification object in the scene image and the coordinates of the identification object in the scene image; inputting the local area image into a MobileFaceNet model to obtain a recognition result of the graphic identification content of the identification object, the graphic identification content including at least one of characters or graphic identifications; and performing an operation in the current scene based on the recognition result of the graphic identification content of the identification object and the coordinates of the identification object in the scene image.

[0004] According to an embodiment of the present invention, a control device based on graphic identification includes: a processor; and a memory on which computer-executable instructions are stored, wherein the computer-executable instructions, when executed by the processor, prompt the processor to execute the above-mentioned control method based on graphic identification.

[0005] According to the computer-readable storage medium of the embodiment of the present invention, computer-executable instructions are stored thereon, wherein when these computer-executable instructions are executed by a processor, the processor is prompted to execute the above-mentioned control method based on graphic and text identification.

[0006] The computer program product according to an embodiment of the present invention includes computer-executable instructions, wherein when these computer-executable instructions are executed by a processor, they prompt the processor to execute the above-mentioned control method based on graphic and text mark recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The present invention can be better understood from the following description of specific embodiments of the present invention in conjunction with the accompanying drawings, in which:

[0008] Figure 1 A schematic flow chart of a control method based on graphic and text identification according to an embodiment of the present invention is shown.

[0009] Figure 2A schematic information flow chart of a control method based on graphic and text identification according to an embodiment of the present invention is shown.

[0010] Figure 3 A schematic diagram of a computer system is shown that can implement the control method and device based on graphic and text mark recognition according to an embodiment of the present invention. DETAILED DESCRIPTION

[0011] The features and exemplary embodiments of various aspects of the present invention will be described in detail below. In the detailed description below, many specific details are proposed to provide a comprehensive understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention can be implemented without the need for some of these specific details. The following description of the embodiments is merely intended to provide a better understanding of the present invention by illustrating examples of the present invention. The present invention is by no means limited to any specific configuration and algorithm proposed below, but covers any modification, replacement, and improvement of elements, components, and algorithms without departing from the spirit of the present invention. In the accompanying drawings and the following description, known structures and techniques are not shown to avoid causing unnecessary ambiguity to the present invention.

[0012] For mobile terminals that need to recognize graphic symbols, such as elevator buttons, the most common recognition algorithm currently used is optical character recognition (OCR). While this algorithm is highly accurate, it requires high image quality. Images currently captured by mobile terminals often contain blurry images and reflective scenes, resulting in poor OCR recognition. Furthermore, the model algorithm is large, requiring high hardware costs.

[0013] In view of the above problems, a control method and device based on graphic identification according to an embodiment of the present invention are proposed, in which the scene image of the current scene is analyzed using the YOLO model and the MobileFaceNet model to obtain the recognition results and coordinates of the graphic identification content of the identification object, which are used to perform operations in the current scene. Combining the YOLO model and the mobile face recognition network (MobileFaceNet) model can better recognize blurred and complex lighting images. The model is also very lightweight and can be adapted to the hardware of mobile terminals at a low cost, taking into account both real-time and accuracy.

[0014] Figure 1 A schematic flow chart of a control method based on graphic and text identification according to an embodiment of the present invention is shown. Figure 2 A schematic information flow chart of a control method based on graphic and text identification according to an embodiment of the present invention is shown. Figure 1 and Figure 2 , a control method based on graphic and text identification according to an embodiment of the present invention is described in detail.

[0015] like Figure 1 As shown, the control method based on graphic identification recognition according to an embodiment of the present invention includes: S101: obtaining a scene image of the current scene; S102: inputting the scene image into the YOLO model to obtain a local area image of the identification object in the scene image and the coordinates of the identification object in the scene image; S103: inputting the local area image into the MobileFaceNet model to obtain the recognition result of the graphic identification content of the identification object, wherein the graphic identification content includes at least one of characters or graphic identifications; S104: performing an operation in the current scene based on the recognition result of the graphic identification content of the identification object and the coordinates of the identification object in the scene image.

[0016] In some embodiments, the scene image of the current scene may be a picture of the current scene captured by a camera, or may be a video frame in a video of the current scene captured by the camera.

[0017] In some embodiments, the YOLO model performs object recognition on the scene image to determine one or more local area images from the scene image. The local area images correspond to identifiable identification objects (e.g., buttons, courier boxes, etc.) in the current scene. The YOLO model can also output the coordinates of the identification objects in the scene coordinates. In some embodiments, the YOLO model can select the YOLOv8m model, which has the advantages of being lightweight and suitable for mobile terminals.

[0018] In the control method according to an embodiment of the present invention, the use of the MobileFaceNet model to classify and identify local area images can reduce the interference of image blur and complex light on image recognition, generate recognition results with higher accuracy, and thus reduce the requirements for image quality. In addition, the recognition result output by the MobileFaceNet model for the identification object is the graphic identification content of the identification object, including at least one of characters (such as Chinese characters, letters, numbers, etc.) or graphic identification (such as elevator door opening signs, elevator door closing signs, wheelchair signs, etc.).

[0019] In some embodiments, after determining the recognition result and coordinates of the identified object, the operation that can be performed in the current scene using the recognition result and coordinates can be a substantial operation on the current scene, such as a substantial operation on the identified object, or a non-substantial operation on the current scene. A substantial operation is, for example, when the identified object is a button (a mechanical button or a button provided on a touch screen), controlling a robotic arm to operate the button, and the operation includes at least one of pressing or touching. A non-substantial operation is, for example, information processing based on the current scene, and the information processing is, for example, adjusting the position of the execution subject (such as an intelligent robot) or adjusting the angle at which the camera captures the scene image based on the recognition result and coordinates, and outputting a scene image that meets the requirements to a specific terminal based on the recognition result and coordinates.

[0020] In the control method of the embodiment of the present invention, the combination of the YOLO model and the MobileFaceNet model can accurately identify graphic and text logos in scene images. Moreover, the model is very lightweight and has a low cost for adapting to mobile terminal hardware. It can be applied to various mobile terminals and other edge devices to achieve graphic and text logo recognition that takes into account both real-time performance and accuracy.

[0021] In some embodiments, the MobileFaceNet model may include one or more attention mechanism modules, each of which is located after a convolution module in the model. The attention mechanism module can be implemented using the SE (Squeeze-and-Excitation) module to weight the channels in the convolutional neural network, so as to enable the network to adaptively adjust the importance of different channels. Specifically, in the MobileFaceNet model, the added SE module can help the network better focus on features that are useful for classification and recognition. The number and position of the SE modules can be dynamically adjusted according to the effect and size of the model. For example, SE modules can be added after the last two convolution modules of MobileFaceNet. Similarly, SE modules can also be added to the YOLO model, specifically after the convolution layer of each CSP (Cross Stage Partial) module of the backbone network of the YOLO model, so that the network can adaptively adjust the weights of different channels.

[0022] Specifically, the SE module includes a squeeze unit, an excitation unit, and a scale unit, where: the squeeze unit obtains the global information of each channel through global average pooling (GAP); the excitation unit uses a fully connected layer to generate channel weights, usually using ReLU activation and Sigmoid activation; the scale unit multiplies the generated weights with the original feature map to adjust the contribution of each channel.

[0023] In some embodiments, the YOLO model and the MobileFaceNet model require pre-training before being used to recognize scene images. For example, the YOLO model is trained using the PyTorch framework. By adjusting the reduction ratio parameter of the SE module in the YOLO model, the degree to which the SE module compresses features can be controlled, ensuring that the SE module meets the model training requirements of this embodiment.

[0024] In some embodiments, the first training set for training the YOLO model and the second training set for training the MobileFaceNet model may include training samples corresponding to multiple lights and multiple blur levels. The training samples in the first training set and the second training set include scene images of multiple scenes as much as possible. Factors that determine each scene include the camera used to capture the image and the captured object. Specifically, the capture posture of the camera may include the capture angle and motion state. The image parameters of the camera may include pixels and resolution, brightness and contrast, frame rate, etc. Different image parameters and capture postures not only determine the image content of the scene image, but also result in different quality of the captured image, that is, different degrees of blur. Since the images captured by the actual camera in motion may be blurred, training the model with training samples of multiple blur levels can ensure the robustness of the model and output accurate recognition results.

[0025] In some embodiments, factors affecting the captured object include the relative position and surface texture of the scene object to be captured, and the angle and intensity of the light projected onto the scene object. When light is projected onto the scene object, it will be reflected based on the surface texture, which will also affect the content of the scene image.

[0026] In some embodiments, when generating the first training set, after acquiring scene images of various scenes, the scene images are processed and annotated to generate training samples. Data augmentation operations may also be performed on the training samples, such as adding different brightness levels, rotating the images, performing perspective transformations, etc., to increase the diversity of the training samples, thereby improving the robustness and generalization ability of the YOLO model.

[0027] In some embodiments, the second training set can be derived from the first training set. Specifically, the images of local regions of the objects annotated in the training samples of the first training set are cropped into small images, and classification labels are added based on the content of the images and text, thereby obtaining the training samples of the second training set. The training samples in the second training set can also be further regularized, for example, by mixing two images and their labels through mixup or randomly occluding parts of the images through cutout, thereby enhancing the robustness of the MobileFaceNet model.

[0028] In some embodiments, the MobileFaceNet model can be trained using a Focal Loss function. This function assigns lower weights to easily classified training samples and emphasizes difficult-to-classify training samples, thereby improving the classification accuracy of the MobileFaceNet model and making it suitable for cases with class imbalance.

[0029] In some embodiments, the MobileFaceNet model can be trained based on the AdamW optimizer, which can introduce weight decay to improve the generalization ability of the model.

[0030] In some embodiments, after completing the training of the YOLO model and the MobileFaceNet model, the YOLO model and the MobileFaceNet model can be quantized according to the configuration requirements of the main control chip of the execution subject of the method of this embodiment (such as an intelligent robot), and then deployed to the main control chip to execute the control method of this embodiment.

[0031] Figure 3 The following is a schematic diagram showing a computer system that can implement the control method and device based on graphic and text identification according to an embodiment of the present invention. Figure 3 The computer system 300 shown is only an example and should not limit the functions and scope of use of the control method and apparatus based on graphic and text identification according to the embodiments of the present invention.

[0032] like Figure 3 As shown, the computer system 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. Various programs and data required for the operation of the computer system 300 are also stored in the RAM 303. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0033] Typically, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touchpad, a camera, an accelerometer, a gyroscope, a sensor, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, a motor, an electronic speed controller, etc.; a storage device 308 including, for example, a flash memory (Flash Card), etc.; and a communication device 309. The communication device 309 may allow the computer system 300 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 3 The computer system 300 is shown with various devices, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have instead. Figure 3 Each block shown in the figure may represent one device, or may represent multiple devices as needed.

[0034] In particular, according to some embodiments of the present invention, the process described above with reference to the flowchart can be implemented as a computer program. For example, a computer readable medium is provided on which a computer program is stored, the computer program including a method for executing Figure 1 The program code of the control method based on graphic and text mark recognition is shown. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functional units defined in the control device based on graphic and text mark recognition according to the embodiment of the present invention are implemented.

[0035] It should be noted that the computer-readable medium according to embodiments of the present invention may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. A computer-readable storage medium according to embodiments of the present invention may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In addition, a computer-readable signal medium according to embodiments of the present invention may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or convey a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0036] Computer program code for performing operations according to embodiments of the present invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect via the Internet).

[0037] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions, and operations of the systems, methods, and computer program products according to the various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0038] The present invention may be implemented in other specific forms without departing from its spirit and essential characteristics. For example, the algorithms described in the specific embodiments may be modified without departing from the basic spirit of the present invention. Therefore, the present embodiments are to be considered in all respects as illustrative and not restrictive, the scope of the invention is defined by the appended claims rather than the foregoing description, and all modifications coming within the meaning and scope of equivalents of the claims are intended to be included within the scope of the present invention.

Claims

1. A control method based on graphic and text identification, comprising: Get the scene image of the current scene; Inputting the scene image into a YOLO model to obtain a local area image of the identified object in the scene image and the coordinates of the identified object in the scene image; Inputting the local area image into a mobile face recognition network (MobileFaceNet) model to obtain a recognition result of the graphic identification content of the identification object, wherein the graphic identification content includes at least one of a character or a graphic identification; as well as An operation is performed in the current scene based on a recognition result of the graphic identification content of the identification object and the coordinates of the identification object in the scene image.

2. The control method according to claim 1, wherein: The YOLO model is a YOLOv8m model.

3. The control method according to claim 1, wherein: The MobileFaceNet model is trained based on the FocalLoss function.

4. The control method according to claim 1, wherein: The MobileFaceNet model is trained based on the AdamW optimizer.

5. The control method according to claim 1, wherein: The MobileFaceNet model includes multiple convolution modules and one or more attention mechanism modules, where the input end of one of the one or more attention mechanism modules is connected to the output end of one of the multiple convolution modules.

6. The control method according to claim 1, wherein: The first training set for training the YOLO model and the second training set for training the MobileFaceNet model both include training samples corresponding to multiple lights and multiple blur levels.

7. The control method according to claim 1, wherein: The identified object is a button, and performing an operation in the current scene includes: controlling a robotic arm to operate the button.

8. A control device based on graphic and text identification, comprising: processor; as well as A memory having computer-executable instructions stored thereon, wherein the computer-executable instructions, when executed by the processor, prompt the processor to execute the control method based on graphic identification according to any one of claims 1 to 7.

9. A computer-readable storage medium having computer-executable instructions stored thereon, wherein: When the computer-executable instructions are executed by a processor, the processor is prompted to execute the control method based on graphic and text identification according to any one of claims 1 to 7.

10. A computer program product comprising computer-executable instructions, wherein: When the computer-executable instructions are executed by a processor, the processor is prompted to execute the control method based on graphic and text identification according to any one of claims 1 to 7.

Images