Display device and text recognition method

Abstract

The application provides a display device and a text recognition method. The display device comprises a display and a controller. After receiving a text recognition instruction input by a user, the display device can generate a text detection box in a user interface for detecting text information in the user interface. In order to avoid the inaccuracy of the text detection box, the text detection box is adjusted to obtain a text region. It should be noted that the text region is a region selected by the adjusted text detection box, and the text region includes a complete piece of text information. The text region is subjected to character recognition processing to obtain text information contained in the user interface. Since each text detection box only contains one piece of text and is complete text information, each piece of text information in the user interface can be accurately recognized, the accuracy of text recognition is improved, and the user experience is strong.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of display devices, and in particular to a display device and a text recognition method. BACKGROUND

[0002] A display device refers to a terminal device capable of outputting a specific display picture, such as a smart television, a mobile terminal, a smart advertising screen, a projector, and the like. With the rapid development of display devices, the functions of display devices will be increasingly rich, and the performance will be increasingly powerful, enabling bidirectional man-machine interaction, integrating audio, entertainment, data, and other functions, and serving to meet the diversified and personalized needs of users.

[0003] With the increasing requirements of users on the functions or services provided by display devices, image recognition has gradually become a function frequently used by users, which can recognize the picture content displayed in a display and provide the text content in the picture to the user. For some display devices, when performing image recognition, a screenshot of the interface currently displayed by the display can be taken, and text recognition can be performed on the screenshot, so as to obtain the text information corresponding to the picture.

[0004] However, in different scenarios, the layout of the text content contained in the picture content displayed by the display device is different, for example, the shape and position of the text content can be different. Therefore, when recognizing the picture content, the region boundary of the text content cannot be accurately determined, resulting in low accuracy of the recognized text content and poor user experience. SUMMARY

[0005] The present application provides a display device and a text recognition method. To solve the problem in the related art that the region boundary of the text content cannot be accurately determined, resulting in low accuracy of the recognized text content and poor user experience.

[0006] In a first aspect, the present application provides a display device, comprising a display and a controller. The display is configured to display a user interface, and the controller is configured to perform the following steps:

[0007] In response to a text recognition instruction input by a user, at least one text detection box is generated in the user interface, the text detection box is used to detect text information in the user interface; the at least one text detection box is adjusted to obtain at least one text region, the text region is a region selected by the adjusted text detection box, and the text region includes a piece of text information; and text recognition processing is performed on the at least one text region to obtain at least one piece of text information.

[0008] In some implementations, the controller is further configured to, in performing the step of generating at least one text detection box in the user interface,

[0009] screenshot processing the user interface to obtain a screenshot image; performing text detection on the screenshot image to obtain at least one text detection box and detection information, the detection information including position information and size information of each text detection box in the user interface.

[0010] In some implementations, the controller is further configured to, before performing the step of adjusting the at least one text detection box,

[0011] determining whether the size information of the text detection box meets a preset size threshold; if yes, performing the step of adjusting the at least one text detection box; if no, deleting the text detection box.

[0012] In some implementations, the controller is further configured to, in performing the step of adjusting the at least one text detection box,

[0013] obtaining a first angle and a second angle of the text detection box according to the position information of the text detection box in the user interface, the first angle and the second angle being angles of two diagonal lines of the text detection box and a horizontal direction, respectively; determining an expansion size of the text detection box according to the first angle and the second angle; and performing expansion processing on the text detection box according to the expansion size.

[0014] In some implementations, the controller is further configured to, in performing the step of adjusting the at least one text detection box,

[0015] obtaining a first text detection box adjacent to the text detection box according to the position information of the text detection box in the user interface; determining whether the text detection box and the first text detection box are of the same type according to a preset text detection box type condition; if they are of the same type, detecting whether a length of the text detection box meets a preset length condition; if yes, not adjusting the text detection box; and if no, adjusting the length of the text detection box according to a preset length formula.

[0016] In some implementations, the controller is further configured to, in performing the step of adjusting the at least one text detection box,

[0017] According to the position information of the text detection box in the user interface, an inclination angle of the text detection box is obtained; when it is detected that the inclination angle is not a preset value, perspective transformation processing is performed on the text detection box to obtain a text region corresponding to the text detection box.

[0018] In some implementations, the controller is further configured to, in the step of performing perspective transformation processing on the text detection box according to the coordinate information,

[0019] Coordinate information of four vertices of the text detection box is obtained; maximum and minimum values of x coordinates and maximum and minimum values of y coordinates in the coordinate information are determined; a perspective transformation matrix is calculated according to the maximum and minimum values of the x coordinates and the maximum and minimum values of the y coordinates; perspective transformation processing is performed on the text detection box according to the perspective transformation matrix to obtain a text region.

[0020] In some implementations, the controller is further configured to, in the step of performing adjustment on the at least one text detection box,

[0021] When it is detected that there is another text detection box in the text detection box, clustering processing is performed on all pixel points in the text detection box to obtain a plurality of types of pixel points, each type of pixel point corresponding to a color; a color is selected to fill the other text detection box existing in the text detection box to obtain a text region corresponding to the text detection box.

[0022] In some implementations, the controller is further configured to, in the step of performing clustering processing on all pixel points in the text detection box,

[0023] A preset number of clustering pixel points are set in the text detection box, each clustering pixel point corresponding to a pixel point type; all pixel points in the text detection box are classified according to the clustering pixel points, each pixel point being classified into a pixel point type corresponding to a nearest clustering pixel point; the clustering pixel points are iteratively updated according to the classification result, and all pixel points are reclassified according to the updated clustering pixel points until convergence, a preset number of pixel point types being obtained, a color corresponding to each pixel point type being a color of the clustering pixel point at the time of convergence.

[0024] In a second aspect, the present application provides a text recognition method applied to a display device, comprising:

[0025] In response to a text recognition instruction input by a user, at least one text detection box is generated in a user interface, the text detection box being used to detect text information in the user interface; the at least one text detection box is adjusted to obtain at least one text region, the text region being a region selected by the adjusted text detection box, and the text region including one piece of text information; and the at least one text region is subjected to character recognition processing to obtain at least one piece of text information.

[0026] As can be seen from the above technical solutions, the present application provides a display device and a text recognition method. After receiving a text recognition instruction input by a user, the display device can generate a text detection box in a user interface, the text detection box being used to detect text information in the user interface. Meanwhile, in order to avoid inaccuracy of the text detection box, the text detection box is also adjusted to obtain a text region. It should be noted that the text region is a region selected by the adjusted text detection box, and the text region includes one complete piece of text information. The text region is subjected to character recognition processing to obtain text information contained in the user interface. Since each text detection box only contains one piece of text and is complete text information, each piece of text information in the user interface can be accurately recognized, the accuracy of text recognition is improved, and the user experience is strong. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the present application, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0028] Figure 1 A use scenario of a display device according to some embodiments is shown;

[0029] Figure 2 A hardware configuration block diagram of a control device 100 according to some embodiments is shown;

[0030] Figure 3 A hardware configuration block diagram of a display device 200 according to some embodiments is shown;

[0031] Figure 4 A software configuration diagram in a display device 200 according to some embodiments is shown;

[0032] Figure 5 A schematic diagram of a user interface in some embodiments is shown;

[0033] Figure 6 A schematic diagram of a screen in a user interface in some embodiments is shown;

[0034] Figure 7Figures showing the display of text recognition mode confirmation information in the display in some embodiments;

[0035] Figure 8 Figures showing the interaction flow of the components of the display device in some embodiments;

[0036] Figure 9 Figures showing the text detection boxes in some embodiments;

[0037] Figure 10 Figures showing the text detection boxes corresponding to the user interface in some embodiments;

[0038] Figure 11 Figures showing the text detection boxes corresponding to the user interface in some embodiments;

[0039] Figure 12 Figures showing the position of the text detection boxes in some embodiments;

[0040] Figure 13 Figures showing the target text detection box and the segmentation area in some embodiments;

[0041] Figure 14 Figures showing the text detection boxes of the user interface in some embodiments;

[0042] Figure 15 Figures showing the position of the adjusted rectangular area in the related art in some embodiments;

[0043] Figure 16 Figures showing the text detection boxes of the user interface in some embodiments;

[0044] Figure 17 Figures showing the text detection boxes of the user interface in some embodiments;

[0045] Figure 18 Figures showing the target text detection box filled with color in some embodiments;

[0046] Figure 19 Figures showing the flow of one embodiment of the text recognition method. DETAILED DESCRIPTION

[0047] In order to make the purposes, embodiments and advantages of the present application clearer, the following will combine the drawings in the exemplary embodiments of the present application to clearly and completely describe the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only some of the embodiments of the present application, but not all the embodiments.

[0048] All other embodiments which would be apparent to one of ordinary skill in the art and that are obtained by combinations of the examples described herein are within the scope of the claims appended hereto. Moreover, although the disclosure is disclosed in terms of one or several examples, it is to be understood that individual aspects disclosed in one example can be combined with aspects described in other examples. It is therefore intended that the description be construed as including all such embodiments and their equivalents. Also, it is intended that the description be construed as including all equivalents to the examples described herein. It is intended that the description be construed as including all examples of the terms described herein. Unless otherwise indicated, such terms are to be construed in their broadest and most general meaning.

[0049] The terms "first", "second", "third", etc. as used in the specification and claims herein and above, are used to distinguish between similar or like objects or entities, and do not necessarily imply a specific order or sequence unless otherwise specified. It is to be understood that the use of these terms in this manner is to be taken as a mere example and the terms are interchangeable under appropriate circumstances.

[0050] The terms "comprise" and "have" and any variations thereof are intended to cover both a complete set of components and an incomplete set of components, for example, a product or device containing a list of components is not necessarily limited to only those components clearly listed, but can include other components not listed or inherent to such products or devices.

[0051] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that can perform the functions described with respect to that element.

[0052] Figure 1 For the purpose of illustrating the operational scenarios between the display device and the control device according to one or more embodiments of the present application, as shown in Figure 1 The user can operate the display device 200 through the mobile terminal 300 and the control device 100. The control device 100 can be a remote controller, and the communication between the remote controller and the display device can include infrared protocol communication, Bluetooth protocol communication, wireless or other wired means to control the display device 200. The user can input user instructions through the buttons on the remote controller, voice input, control panel input, etc. to control the display device 200. In some embodiments, mobile terminals, tablets, computers, laptops, and other smart devices can also be used to control the display device 200.

[0053] In some embodiments, the mobile terminal 300 can install software applications with the display device 200, implement connection communication through network communication protocols, and achieve the purpose of one-to-one control operation and data communication. The mobile terminal 300 can also transmit audio and video content displayed thereon to the display device 200 to achieve a synchronous display function. The display device 200 can also communicate with the server 400 through various communication modes. The display device 200 can be allowed to communicate through a local area network (LAN), a wireless local area network (WLAN), and other networks. The server 400 can provide various content and interaction to the display device 200. The display device 200 can be a liquid crystal display, an OLED display, or a projection display device. In addition to providing a broadcast receiving television function, the display device 200 can also provide a smart network television function with computer support.

[0054] Figure 2 An exemplary configuration block diagram of the control device 100 according to an exemplary embodiment is shown. As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 can receive input operation instructions from a user and convert the operation instructions into instructions that the display device 200 can recognize and respond to, thereby serving as an intermediary between the user and the display device 200. The communication interface 130 is used for external communication and includes at least one of a WIFI chip, a Bluetooth module, an NFC, or an alternative module. The user input / output interface 140 includes at least one of a microphone, a touchpad, a sensor, a key, or an alternative module. Figure 2

[0055] An exemplary hardware configuration block diagram of the display device 200 according to an exemplary embodiment is shown. As shown, the display device 200 includes a controller 210, a communication interface 230, a user input / output interface 240, a memory, and a power supply. The display device 200 can receive input operation instructions from a user and convert the operation instructions into instructions that the control device 100 can recognize and respond to, thereby serving as an intermediary between the user and the control device 100. The communication interface 230 is used for external communication and includes at least one of a WIFI chip, a Bluetooth module, an NFC, or an alternative module. The user input / output interface 240 includes at least one of a microphone, a touchpad, a sensor, a key, or an alternative module. Figure 3 Figure 3 ​The display device 200 includes at least one of a tuner and demodulator 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a storage, a power supply, and a user interface 280. The controller includes a central processing unit, a video processor, an audio processor, a graphic processor, a RAM, a ROM, and a first interface to an n-th interface for input / output. The display 260 can be at least one of a liquid crystal display, an OLED display, a touch display, and a projection display, and can also be a projection device and a projection screen. The tuner and demodulator 210 receives a broadcast television signal through wired or wireless reception, and demodulates audio and video signals and EPG data signals from a plurality of wireless or wired broadcast television signals. The detector 230 is configured to collect signals of an external environment or interaction with the external environment. The controller 250 and the tuner and demodulator 210 can be located in different split devices, i.e., the tuner and demodulator 210 can also be located in an external device of the main body device where the controller 250 is located, such as an external set-top box.

[0056] In some embodiments, the controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in the storage. The controller 250 controls the overall operation of the display device 200. The user can input a user command through a graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user can input a user command by inputting a specific sound or gesture, and the user input interface receives the user input command by recognizing the sound or gesture through a sensor.

[0057] In some embodiments, the "user interface" is a medium interface for interaction and information exchange between an application program or an operating system and a user, which realizes the conversion between an internal form of information and a form acceptable by the user. A commonly used form of the user interface is a graphical user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be an icon, a window, a control, etc. displayed on the display screen of an electronic device, wherein the control can include at least one of an icon, a button, a menu, a tab, a text box, a dialog box, a status bar, a navigation bar, a widget, and other visible interface elements.

[0058] Figure 4 A software configuration diagram of the display device 200 according to one or more embodiments of the present application is shown in FIG. 2. The display device 200 includes a tuner and demodulator 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a storage, a power supply, and a user interface 280. Figure 4As shown, the system is divided into four layers from top to bottom, namely, an Applications layer (referred to as an "application layer"), an Application Framework layer (referred to as a "framework layer"), an Android runtime and system library layer (referred to as a "system runtime library layer"), and a kernel layer. The kernel layer at least includes at least one of the following drivers: an audio driver, a display driver, a Bluetooth driver, a camera driver, a WIFI driver, a USB driver, an HDMI driver, a sensor driver (such as a fingerprint sensor, a temperature sensor, a pressure sensor, etc.), and a power supply driver, etc.

[0059] The display device can have an image recognition function, which can recognize the picture content displayed in the display and provide the text content in the picture to the user. For some display devices, when performing image recognition, a screenshot of the interface currently displayed by the display can be taken, and text recognition is performed on the screenshot to obtain the text information corresponding to the picture.

[0060] However, in different scenarios, the layout of the text content contained in the picture content displayed by the display device is different, for example, the shape and position of the text content can be different. Therefore, when recognizing the picture content, the boundary of the text content area cannot be accurately determined, resulting in low accuracy of the recognized text content and poor user experience.

[0061] The present application provides a display device, which includes a display and a controller. The display is used to display a user interface. The user interface can be the picture content currently displayed in the display. For example, the user interface can be a specific target image, such as various media obtained from a network signal source, including video, picture, etc. The user interface can also be some UI interface of the display device.

[0062] Figure 5 Some schematic diagrams of the user interface in some embodiments are shown. The user interface includes a first navigation bar 500, a second navigation bar 510, a function bar 520, and a content display area 530. The function bar 520 includes a plurality of function controls such as "view record", "my collection", and "my application", etc. The content displayed in the content display area 530 will change with the change of the selected control in the first navigation bar 500 and the second navigation bar 510. When the application panel page, the user can input a display instruction for the application panel page by clicking the "my application" control to trigger entering the corresponding application panel. It should be noted that the user can also input the selection operation of the function control by other ways to trigger entering the application panel. For example, using the voice control function or the search function, etc., to control entering the application panel page.

[0063] In some embodiments, the user interface displayed in the display can include a media area and a text area. The media area is used to display specific media, such as a video of a TV series, a movie, or some pictures. The text area is specific text information, such as the names of various controls and prompt words. The user interface can include multiple text areas, and each text area can include a complete text content. For example, Figure 5 The control names "AI home" and "channel" in the above figure can be regarded as a text area.

[0064] In some embodiments, the user interface can display a media recommendation control. The media recommendation control can be an introduction picture of a media resource, Figure 6 The above figure shows a screen diagram of a user interface in some embodiments. The media recommendation control can include a media picture and a media text on the picture, and the media text is "Movie A" in the above figure. The media text in the media recommendation control can also be regarded as a text area. Figure 6

[0065] By performing text recognition on the text area, the text content contained in the text area can be obtained. It should be noted that the text area is different for different user interfaces, and therefore, the text area in the user interface needs to be accurately recognized, so that the text content in the user interface can be recognized.

[0066] In some embodiments, the display device has a text recognition function, and can recognize a target control that needs to be triggered by a user according to a user input instruction. The display device can be provided with a text recognition mode. In the text recognition mode, the display device can automatically recognize all controls in the user interface, so as to obtain the target control that needs to be triggered by the user.

[0067] In some embodiments, the user can send a text recognition mode instruction to the display device by operating a specified key of the remote controller. The correspondence between the text recognition mode instruction and the remote controller key is bound in advance in actual application. For example, a text recognition mode key is provided on the remote controller. When the user touches the key, the remote controller sends the text recognition mode instruction to the controller, and the controller controls the display device to enter the text recognition mode. When the user touches the key again, the controller can control the display device to exit the text recognition mode.

[0068] ​In some embodiments, the correspondence between the text recognition mode instruction and the plurality of remote controller keys can also be bound in advance. When the user touches the plurality of keys bound with the text recognition mode instruction, the remote controller sends the text recognition mode instruction. In a feasible embodiment, the keys bound with the text recognition mode instruction are the direction keys (left, down, left, down) in sequence. That is, when the user successively touches the keys (left, down, left, down) within a preset time, the remote controller sends the text recognition mode instruction to the controller. By using the above binding method, the text recognition mode instruction can be prevented from being sent due to user's misoperation. The embodiments of the present application only exemplarily provide several binding relationships between the text recognition mode instruction and the keys. In actual application, the binding relationship between the text recognition mode instruction and the keys can be set according to user's habits, which is not limited herein.

[0069] In some embodiments, the user can use the sound collector of the display device, such as a microphone, to send the text recognition mode instruction to the display device by voice input, so as to control the display device to enter the text recognition mode. The display device can be provided with a smart voice system. The smart voice system can recognize the user's voice to extract the instruction content input by the user. The user can input a preset wake-up word through the microphone, so as to start the smart voice system, so that the controller can respond to the instruction input by the user. And input the text recognition mode instruction within a certain time, so that the display device enters the text recognition mode. For example, the user can input "Xiao Xuesheng" to start the smart voice system. Then input "enter the text recognition mode" to send the text recognition mode instruction to the display device.

[0070] In some embodiments, the user can also send the text recognition mode instruction to the display device through a preset gesture. The display device can detect the user's behavior through an image collector, such as a camera. When the user makes a preset gesture, it can be considered that the user sends the text recognition mode instruction to the display device. For example, it can be set that when the user draws a V shape, it is determined that the user inputs the text recognition mode instruction to the display device. The user can also send the text recognition mode instruction to the display device through a preset action. For example, it can be set that when the user simultaneously lifts the left foot and the right hand, it is determined that the user inputs the text recognition mode instruction to the display device.

[0071] In some embodiments, when the user uses the smart device to control the display device, such as using a mobile phone, the text recognition mode instruction can also be sent to the display device. In actual application, a control can be set in the mobile phone. Whether to enter the text recognition mode can be selected through the control, so as to send the text recognition mode instruction to the controller. At this time, the controller can control the display device to enter the text recognition mode.

[0072] In some embodiments, when the user uses the mobile phone to control the display device, the mobile phone can be instructed to perform continuous clicking. The continuous clicking instruction refers to that the user clicks the same area of the touch screen of the mobile phone more than a preset threshold number of times within a preset period. For example, when the user clicks the same area of the touch screen of the mobile phone 3 times within 1 second, it is considered as one continuous clicking instruction. After receiving the continuous clicking instruction, the mobile phone can send a text recognition mode instruction to the display device, so that the controller controls the display device to enter the text recognition mode.

[0073] In some embodiments, when the user uses the mobile phone to control the display device, the mobile phone can also be instructed to send a text recognition mode instruction to the display device when it is detected that the user touches the touch screen of the mobile phone with a pressure value exceeding a preset pressure threshold.

[0074] The text recognition mode option can also be set in the UI interface of the display device. When the user clicks the option, the display device can be controlled to enter or exit the text recognition mode.

[0075] In some embodiments, to prevent the user from triggering the text recognition mode by mistake, when the controller receives the text recognition mode instruction, the display can be controlled to display text recognition mode confirmation information, so that the user makes a second confirmation as to whether to control the display device to enter the text recognition mode. Figure 7 A schematic diagram showing the display of the text recognition mode confirmation information in the display of some embodiments is shown.

[0076] When the display device enters the text recognition mode, the current user interface can be automatically subjected to text recognition, so as to recognize the text content contained in the user interface.

[0077] Figure 8 A flowchart showing the interaction of the components of the display device in some embodiments is shown.

[0078] In some embodiments, when the user controls the display device to start, the controller can control the display to display a user interface. When the user controls the display device to enter the text recognition mode, the text content contained in the user interface can be recognized. The user can send a text recognition instruction to the display device. The text recognition instruction is used to instruct the display device to enter the text recognition mode, i.e., to instruct the display device to perform text recognition on the user interface.

[0079] When the display device receives the text recognition instruction sent by the user, the controller can perform text recognition processing on the user interface to obtain the text content contained in the user interface.

[0080] In some embodiments, when the text recognition instruction sent by the user is received, the controller can first determine the screen displayed in the display, i.e., the user interface.

[0081] The controller can first perform screenshot processing on the user interface to obtain a screenshot image of the user interface. Specifically, the controller can perform screenshot processing on the current display in the display through a screenshot program to obtain a screenshot image of the user interface. After obtaining the screenshot image of the user interface, the controller can further detect the screenshot image to determine the text information contained in the screenshot image, that is, the text information contained in the user interface.

[0082] Specifically, the controller can first determine the text region contained in the screenshot image, and then obtain the text information contained in each text region according to each text region, so as to accurately identify the text information in the user interface.

[0083] The controller can perform text detection on the screenshot image, determine the text region contained in the user interface by using a text box detection method, that is, select the text region by using a text detection box. Specifically, the controller can generate a text detection box in the user interface, that is, in the screenshot image. The text detection box can be used to detect the text region contained in the screenshot image, that is, the text detection box can detect the text information in the user interface. For the user interface shown in FIG. 6A, after text detection, a schematic diagram of a plurality of text detection boxes can be obtained, as shown in FIG. 6B. Among them, the control text and the media text can each correspond to a text detection box. Figure 6 The controller can perform text detection on the screenshot image, determine the text region contained in the user interface by using a text box detection method, that is, select the text region by using a text detection box. Specifically, the controller can generate a text detection box in the user interface, that is, in the screenshot image. The text detection box can be used to detect the text region contained in the screenshot image, that is, the text detection box can detect the text information in the user interface. For the user interface shown in FIG. 6A, after text detection, a schematic diagram of a plurality of text detection boxes can be obtained, as shown in FIG. 6B. Among them, the control text and the media text can each correspond to a text detection box. Figure 9

[0084] In some embodiments, when the controller performs text detection on the screenshot image, the controller can use an OCR (Optical Character Recognition) technology to detect the screenshot image, thereby generating at least one text detection box in the screenshot image to determine at least one text region contained in the screenshot image. For example, the OCR technology can be used to perform text detection on the screenshot image to obtain a plurality of corresponding text detection boxes. The OCR method can determine the shape of the text region by detecting the dark and bright patterns, and then generate the corresponding text detection box, which is a prior art. The specific OCR recognition method can be set according to the hardware configuration of the actual display device, for example, the recognition method can be based on artificial intelligence, neural network, genetic algorithm, etc. The embodiments of the present application are not limited. Other text recognition technologies can also be used to process the screenshot image. The embodiments of the present application are described by way of example using the OCR technology, but are not limited in specific.

[0085] ​When the screenshot image is detected by using the OCR technology, the text detection box can be generated in the screenshot image, and the detection information of each text detection box can also be obtained. Specifically, for each text detection box, the detection information can include the position information of the text detection box in the user interface and the size information of the text detection box.

[0086] Specifically, the position information of the text detection box in the user interface can be the coordinate information of four vertices of the text detection box. For example, for the text detection box L, the four vertices include a left upper corner vertex A, a left lower corner vertex B, a right upper corner vertex C, and a right lower corner vertex D. In the embodiments of the present application, the coordinate system is established with the left upper corner vertex of the display as the origin, the horizontal right direction as the positive direction of the x-axis, and the vertical downward direction as the positive direction of the y-axis. The position information of the text detection box L is: A coordinate (x1, y1), B coordinate (x4, y4), C coordinate (x2, y2), and D coordinate (x3, y3). The size information of the text detection box includes length information and height information. For example, for the text detection box L, the length is W and the height is h.

[0087] In some embodiments, if the text detection box L is a rectangular region without an inclination angle, the detection information can also be uniformly represented, that is, the position information and the size information are uniformly represented. For example, the detection information can be represented in the form of (x, y, W, h). Wherein (x, y) represents the coordinates of a vertex or a center point of the text detection box, W represents the length of the text detection box, and h represents the height of the text detection box.

[0088] In some embodiments, by performing text detection on the screenshot image of the user interface, a plurality of text detection boxes corresponding to the screenshot image and the detection information of each text detection box can be obtained.

[0089] It should be noted that for different user interfaces, the format of the text region, such as shape and position, is different, and therefore the text detection box obtained by performing text detection on the screenshot image can not accurately detect the text region. For example, the text detection box can be too small, resulting in that the complete text region is not selected. Figure 10 A schematic diagram of the text detection box corresponding to the user interface in some embodiments is shown. Among them, the text detection box corresponding to the media text "Movie A" does not select the media text "Movie A" at all. Figure 11 A schematic diagram of the text detection box corresponding to the user interface in some embodiments is shown. Among them, the control texts "Movie" and "TV series" are selected by a text detection box at the same time, resulting in an inaccurate recognition result.

[0090] Therefore, after text detection is performed on the screenshot image to obtain a plurality of text detection boxes, the text detection boxes need to be adjusted to ensure that each adjusted text detection box can select a complete text region, i.e., the region selected by the adjusted text detection box is a text region. At this time, each adjusted text detection box can include a complete text information.

[0091] In some embodiments, considering that some shapes such as symbols or icons can be mistaken for text content, a smaller text detection box is generated, which affects the recognition of the text content. Therefore, a size condition can be set for the text detection box. When the size of a text detection box does not meet the size condition, it indicates that the text detection box is small, and the region selected by the text detection box is not text content. At this time, the text detection box should be ignored.

[0092] The size condition can be set as that the text detection box meets a preset size threshold. The size threshold can be set as that the length of the text detection box is p and the height is q. That is, for a text detection box, when the length is ≥ p and the height is ≥ q, it indicates that the text detection box meets the size threshold, i.e., meets the size condition.

[0093] Specifically, the controller can first determine the size information of each text detection box and determine whether the size information of the text detection box meets the preset size threshold. If yes, it indicates that the region selected by the text detection box includes a text region. At this time, the text detection box can be continuously adjusted to ensure that each adjusted text detection box can select a complete text region. If not, it indicates that the region selected by the text detection box is not text content. At this time, the text detection box can be deleted.

[0094] In some embodiments, when the text detection box is adjusted, for some text detection boxes as shown in FIG. 6, i.e., the text detection box is too small to select a complete text region, the controller can adjust the text detection box, expand the text detection box, and increase the size of the text detection box to enable the text detection box to select a complete text region. Figure 10

[0095] In some embodiments, when the text detection box is expanded, the text detection box can be expanded by a preset number of pixel points along the diagonal direction, which can be 2-4 pixel points. For example, the four vertices of the text detection box can be expanded by 4 pixel points along the respective diagonal directions. By expanding the text detection box, the region selected by the text detection box can be a complete text region.

[0096] ​In some embodiments, considering that the text detection box can be in a tilted state, therefore when the text detection box is processed for expansion, the size of the text detection box expansion can be adjusted according to the tilted state of the text detection box.

[0097] The controller can first obtain a first angle and a second angle of the text detection box, the first angle and the second angle referring to the included angle of two diagonal lines of the text detection box and the horizontal direction. Specifically, the first angle θ and the second angle ψ can be calculated according to the position information of the text detection box in the user interface.

[0098] For the text detection box L, the position information is: the top-left corner vertex A(x1, y1), the bottom-left corner vertex B(x4, y4), the top-right corner vertex C(x2, y2), and the bottom-right corner vertex D(x3, y3). Figure 12 Some embodiments of the position diagram of the text detection box are shown.

[0099] The calculation method of the first angle θ and the second angle ψ of the text detection box can be:

[0100]

[0101]

[0102]

[0103]

[0104] After the first angle θ and the second angle ψ of the text detection box are calculated, the size of the text detection box expansion can be calculated according to the two angles, and the calculation formula can be:

[0105]

[0106]

[0107]

[0108]

[0109]

[0110]

[0111] D1 and D2 are the sizes of the text detection box in the diagonal direction, and a and b are preset parameters. For example, a can be 0.1, and b can be 0.15.

[0112] By expanding the text detection box, the region selected by the text detection box can be a complete text region, so that the text content in the user interface can be accurately recognized.

[0113] In some embodiments, when adjusting the text detection box, for some text detection boxes as shown in FIG. 5B, that is, a plurality of text regions are selected in one text detection box, the controller can adjust the text detection box so that only one text region is selected in each text detection box, so that the accurate text content can be recognized. Figure 11

[0114] It should be noted that when the screenshot image of the user interface is subjected to text detection, for the same type of text region, the same text detection box may select multiple text regions. As shown in FIG. 5B, for each control in the first navigation bar 500, the text region corresponding to the control is a text region of the same type, so when text detection is performed, a text detection box may select multiple text regions. The text content finally recognized by such a text detection box is obviously inaccurate text content, so the text detection box needs to be adjusted so that only one text region is selected in each text detection box. Figure 5

[0115] When adjusting the text detection box, the controller can first determine whether multiple text regions are selected in the text detection box. When multiple text regions are selected in the text detection box, the text detection box can be further adjusted.

[0116] The controller can first detect whether there is a text detection box of the same type around the target text detection box to be adjusted. When there is a text detection box of the same type around the target text detection box, it means that the text type regions selected by the two text detection boxes are the same, that is, there are multiple text regions of the same type around the target text detection box, and the target text detection box may also select multiple text regions of the same type.

[0117] In some embodiments, the text detection boxes of the same type refer to that the overall heights of the two text detection boxes are substantially the same, and the two text detection boxes are substantially located on a horizontal line.

[0118] ​​In order to determine whether there is a text detection box of the same type around the target text detection box, the controller can first obtain a text detection box adjacent to the target text detection box. Specifically, a first text detection box adjacent to the target text detection box can be determined according to the position information of the text detection box. In the embodiments of the present application, the text detection box adjacent to the target text detection box refers to a text detection box that is located on substantially the same horizontal line as the target text detection box and is closest to the target text detection box.

[0119] Specifically, in determining the first text detection box, the controller can first detect all text detection boxes to obtain all text detection boxes that are located on substantially the same horizontal line as the target text detection box. A detection condition can be set: Δy < y0. Wherein, Δy represents the difference between the y coordinates of the top left corners of two text detection boxes, and y0 represents a preset y coordinate difference threshold. When the difference between the y coordinates of the top left corners of two text detection boxes is within the preset y coordinate difference threshold, it indicates that the two text detection boxes are located on substantially the same horizontal line. According to the detection condition, all text detection boxes that are located on substantially the same horizontal line as the target text detection box can be obtained. The controller then detects the difference between the x coordinates of the top left corners of these text detection boxes and the top left corner of the target text detection box, and the text detection box corresponding to the minimum difference is the first text detection box.

[0120] The position information of the first text detection box is set as: the top left corner (x1*, y1*), the bottom left corner (x4*, y4*), the top right corner (x2*, y2*) and the bottom right corner (x3*, y3*). The length of the first text detection box is W* and the height is h*.

[0121] After determining the first text detection box adjacent to the target text detection box, the controller can detect whether the target text detection box and the first text detection box are of the same type. Specifically, whether the two text detection boxes are of the same type can be determined according to a preset text detection box type condition.

[0122] The preset text detection box type condition can be: abs|h-h*| < h0. Wherein, abs|h-h*| represents the height difference between two text detection boxes, and h0 represents a preset height difference threshold. When the height difference between two text detection boxes is within the preset height difference threshold, it indicates that the overall height of the two text detection boxes is substantially the same. At this time, the two text detection boxes are of the same type.

[0123] In some embodiments, the first text bounding box can also not be acquired. The controller can detect all text bounding boxes under the condition that abs|y1-y1*|<y0 and abs|h-h*|<h0. When both conditions are met, it means that the text bounding box and the target text bounding box are of the same type, i.e., the target text bounding box exists in the same type of text bounding box.

[0124] In some embodiments, when it is determined that the same type of text bounding box exists around the target text bounding box, the controller can detect whether multiple text regions are selected in the target text bounding box.

[0125] Specifically, the controller can detect whether the length of the target text bounding box meets a preset length condition.

[0126] The preset length condition can be abs|W-W*|<W*. Wherein, abs|W-W*| represents the length difference of the two text bounding boxes. The length condition indicates that when the length difference between the target text bounding box and the first text bounding box does not exceed the length of the first text bounding box, it means that the target text bounding box does not select multiple text regions.

[0127] If it is detected that the length of the target text bounding box meets the preset length condition, the target text bounding box does not select multiple text regions, at this time the target text bounding box can not be adjusted, or other adjustment methods can be used to adjust the target text bounding box.

[0128] If it is detected that the length of the target text bounding box does not meet the preset length condition, the target text bounding box selects multiple text regions, at this time the target text bounding box needs to be adjusted. Specifically, a preset length formula can be used to adjust the length of the text bounding box, and the length of the text bounding box is reduced. The preset length formula is x2=x1+2W* and x3=x4+2W*. That is, the top-left corner vertex and the bottom-left corner vertex are kept unchanged, the x-coordinate x2 of the top-right corner vertex is changed to x1+2W*, and the x-coordinate x3 of the bottom-right corner vertex is changed to x4+2W*. Thus, the length of the target text bounding box is reduced as a whole, so that the target text bounding box selects one text region.

[0129] In some embodiments, if it is detected that the length of the target text bounding box does not meet the preset length condition, i.e., the target text bounding box selects multiple text regions. In order to preserve these multiple text regions, the controller can perform segmentation processing on the target text bounding box to obtain multiple text bounding boxes, so that each text bounding box contains one text region.

[0130] Specifically, for the target text bounding box L, the length is W and the height is h. The position information of the target text bounding box L is: the top-left corner vertex A (x1, y1), the bottom-left corner vertex B (x4, y4), the top-right corner vertex C (x2, y2), and the bottom-right corner vertex D (x3, y3). When multiple text regions are selected in the target text bounding box L, the target text bounding box L can be segmented by using the segmentation region. The segmentation region can be a part of the target text bounding box L, and the height of the segmentation region is set to h and the length is set to s. That is, the target text bounding box L is segmented by using the segmentation region which is equal in height to the target text bounding box L and located inside the target text bounding box L, so as to segment the target text bounding box L into multiple text bounding boxes.

[0131] In some embodiments, when the screenshot image of the user interface is subjected to text detection, although the accuracy of text detection can be low, multiple text regions can be selected in the obtained text detection box, but generally only two text regions are selected. The middle part of the two text regions is a space region, that is, the background region of the user interface, and this part of the space region can affect text recognition. Therefore, it is necessary to eliminate this part of the space region.

[0132] Specifically, the target text bounding box L can be segmented into two text bounding boxes by using the segmentation region. The segmentation region is equal in height to the target text bounding box L, and the height is h. The length s of the segmentation region can be set to 0.2W.

[0133] Figure 13 A schematic diagram of the target text bounding box and the segmentation region in some embodiments is shown. First, the left side of the segmentation region is located at W / 3 in the length direction of the target text bounding box L, that is, the left side of the segmentation region is 1 / 3 of the target text bounding box L. At this time, the position information of the left side of the segmentation region is: the top-left corner vertex

x1+(x2-x1) / 3, y1-(y1-y2) / 3

x4+(x3-x4) / 3, y4-(y4-y3) / 3

[0134] The controller can detect whether the pixel values of all the pixel points in the segmentation region are the same. If they are the same, it means that there is no character in the segmentation region, that is, there is no text region. At this time, the target text bounding box L can be segmented by using the segmentation region to obtain two text bounding boxes, and each text bounding box corresponds to a text region.

[0135] If they are different, it indicates the presence of a text region within the segmentation area, requiring adjustment of the segmentation area's position. Specifically, the size of the segmentation area can be kept constant, and it can be shifted 0.1W units to the right along the length of the target text detection box L. After obtaining the shifted segmentation area, the controller can continue to check if the pixel values ​​of all pixels within the current segmentation area are the same. If they are the same, the target text detection box L is segmented; otherwise, the segmentation area is shifted until the pixel values ​​of all pixels within the segmentation area are the same, thus segmenting the target text detection box L and obtaining two text detection boxes.

[0136] It should be noted that when there are two text regions within the target text detection box L, the middle part of the two text regions is a blank area, which can be considered as the middle area of ​​the target text detection box L being a blank area. Therefore, a range can be set for the segmentation region, which can be set as follows: the left edge of the segmentation region is located between W / 3 and 2W / 3 of the length direction of the target text detection box L, thereby reducing the amount of computation.

[0137] In some embodiments, the controller can first segment the target text detection box, and then adjust the length of the segmented text detection boxes. When adjusting the length of the text detection boxes, preset length conditions and preset length formulas can be used. For specific adjustment methods, please refer to the foregoing content, which will not be repeated here.

[0138] In some embodiments, considering such Figure 5 The multiple vertical text areas shown in the second navigation bar 510 are also of the same type. Therefore, when adjusting the text detection boxes, text detection boxes of the same type can be set as follows: the overall length of the two text detection boxes is approximately the same, and the two text detection boxes are approximately located on a vertical line. In this case, when the difference in the x-coordinates of the top-left corners of the two text detection boxes is within a preset x-coordinate difference threshold, it indicates that the two text detection boxes are approximately located on a vertical line. When the length difference of the two text detection boxes is within a preset length difference threshold, it indicates that the overall length of the two text detection boxes is approximately the same. In this case, the two text detection boxes are of the same type.

[0139] When two text detection boxes are determined to be of the same type, it can be checked whether the height of the target text detection box meets a preset height condition. The preset height condition means that if the height difference between the target text detection box and the first text detection box does not exceed the height of the first text detection box, it indicates that the target text detection box has not selected multiple text areas.

[0140] If it is detected that the target text detection box meets the height condition, the target text detection box can not be adjusted. If it is detected that the target text detection box does not meet the height condition, a preset height formula can be used to adjust the height of the text detection box, and the height of the text detection box is reduced. The preset height formula is: y4=y1+2h* and y3=y2+2h*.

[0141] In some embodiments, when the related art identifies the text content in the text detection box, if the text detection box is in an inclined state, a complete rectangular region without an inclination angle needs to be intercepted for character recognition. Figure 14 A schematic diagram of a text detection box of a user interface in some embodiments is shown. The user interface includes text detection boxes L and L1. For the text detection box L, the left upper corner vertex A (x1, y1), the left lower corner vertex B (x4, y4), the right upper corner vertex C (x2, y2), and the right lower corner vertex D (x3, y3), where x4 Figure 15 A schematic diagram of the position of the adjusted rectangular region L* in the related art in some embodiments is shown. The position information of L* is: the left upper corner vertex (x4, y1), the left lower corner vertex (x4, y3), the right upper corner vertex (x2, y1), and the right lower corner vertex (x2, y3). However, the rectangular region thus determined will contain the text detection box L1, i.e., other text content, resulting in errors in the recognized text content and a low accuracy. Therefore, when the text detection box is in an inclined state, the text detection box needs to be adjusted.

[0142] The controller can first detect whether the text detection box is in an inclined state. The detection can be whether the inclination angle of the text detection box is a preset value. The inclination angle refers to the angle between the edge of the text detection box and the horizontal direction. In the embodiments of the present application, the preset value can be 0° or 90°. When it is detected that the inclination angle is the preset value 0° or 90°, it is indicated that the text detection box is not in an inclined state. When it is detected that the inclination angle is not the preset value 0° or 90°, it is indicated that the text detection box is in an inclined state.

[0143] When it is detected that the text detection box is in a tilted state, the controller can perform perspective transformation processing on the text detection box, so as to ensure that the text detection box does not contain other text regions. It should be noted that perspective transformation projects a picture to a new view plane, also known as projection mapping. By using the condition that the perspective center, image point and target point are collinear, the projection surface (perspective surface) is rotated by a certain angle according to the law of perspective rotation, the original projection light beam is destroyed, and the projection geometry on the projection surface remains unchanged. This is the prior art.

[0144] Specifically, when performing perspective transformation processing on the text detection box, the controller first determines the maximum and minimum values of the x coordinates and the maximum and minimum values of the y coordinates of the four vertices of the text detection box according to the coordinate information of the four vertices. The perspective transformation matrix can be calculated according to the maximum and minimum values of the x coordinates and the maximum and minimum values of the y coordinates. Further, the text detection box can be processed by perspective transformation according to the perspective transformation matrix, so as to obtain an adjusted text detection box. The selected region in the adjusted text detection box is a text region.

[0145] In some embodiments, the text detection boxes can be nested or intersected. Figure 16 A schematic diagram of the text detection boxes of the user interface in some embodiments is shown. The text content in the text detection box L is "Twelve Zodiac", and the text content in the text detection box L1 is "Twelve". However, the text detection box L1 is located in the text detection box L, forming a nested condition. When performing character recognition on the text detection box L, it is possible to recognize the text content in the text detection box L1, which can cause the recognition result of the text detection box L to be inaccurate.

[0146] Figure 17 A schematic diagram of the text detection boxes of the user interface in some embodiments is shown. The text content in the text detection box L is "Twelve Zodiac", and the text content in the text detection box L2 is "Coming soon". However, the text detection box L and the text detection box L2 intersect. At this time, when performing character recognition on the text detection box L, it is also possible to recognize the text content "Coming soon" in the text detection box L2, which can cause the recognized text content to be incorrect, seriously affecting the recognition result and causing the accuracy to be low. Therefore, the text detection box needs to be adjusted.

[0147] When it is detected that there are other text detection boxes in the text detection box, including the text detection box nesting condition and the text detection box intersection condition, the controller needs to adjust the target text detection box to be detected. The controller can eliminate the other text detection boxes existing in the target text detection box, so as to ensure that the target text detection box does not contain the text content in the other text detection boxes.

[0148] Specifically, the controller can perform clustering processing, such as K-means clustering processing or hierarchical clustering, on all the pixel points in the target text detection box. After clustering processing is performed on all the pixel points, all the pixel points can be divided into n types, and each type of pixel point corresponds to a separate color, that is, the clustering result includes n colors.

[0149] The controller can select one of the colors to fill other text detection boxes present in the target text detection box, thereby eliminating all other text detection boxes and ensuring that the target text detection box selects a text region.

[0150] In some embodiments, the controller can first set a predetermined number of clustering pixel points, such as four clustering pixel points, in the target text detection box. The four clustering pixel points can be randomly selected as initial clustering pixel points in the target text detection box, or the four initial clustering pixel points can be pre-set by a designer. Each clustering pixel point corresponds to a pixel point type, and each pixel point type corresponds to a color.

[0151] After the four clustering pixel points are determined, the controller can calculate the distance from all the pixel points in the target text detection box to each clustering pixel point. At the same time, the principle of near distance classification is adopted to classify each pixel point into the pixel point type corresponding to the nearest clustering pixel point. According to the principle of near distance classification, all the pixel points can be classified into four pixel point types, achieving the classification of all the pixel points in the target text detection box. Alternatively, the similarity (Euclidean distance) of all the pixel points to each clustering pixel point can be calculated to measure the absolute deviation or squared deviation between a pixel point and a clustering pixel point, and the pixel points can be classified into the pixel point type corresponding to the clustering pixel point with the smallest similarity.

[0152] To address the inaccuracy of sequential classification, clustering processing can also be used to continuously classify all the pixel points. Specifically, the four clustering pixel points can be re-determined according to the classification result of the previous time, achieving the update of the clustering pixel points.

[0153] When updating the clustering pixel points, for each clustering pixel point, the mean value of all the pixel points of the corresponding pixel point type can be determined according to the classification result, that is, the mean value of the x coordinates and the mean value of the y coordinates of all the pixel points are determined. The pixel point at the coordinates corresponding to the mean value of the x coordinates and the mean value of the y coordinates is taken as a new clustering pixel point, achieving the update of the clustering pixel points.

[0154] Further, the classification of all the pixel points can be continued according to the updated clustering pixel points. By continuously classifying the pixel points and iteratively updating the clustering pixel points, until the clustering converges (convergence refers to the distance changed after iteration of each clustering pixel point is less than a preset distance, which can be 1) or reaches a preset maximum iteration number (which can be 20 times), at this time, the final four clustering pixel points can be obtained. All the pixel points in the target text detection box are assigned to four pixel point types, and the color corresponding to each pixel point type is the color of the clustering pixel point at the time of convergence.

[0155] At this time, the controller can select a color to fill other text detection boxes.

[0156] In some embodiments, after obtaining the color corresponding to each pixel point type, the colors can be sorted. Specifically, the pixel points in each color corresponding to the pixel point type can be sorted according to the number of pixel points, for example, sorted in descending order of the number. Among them, the color corresponding to the largest number of pixel points can be considered as the text color of the text content in the target text detection box. At the same time, a color different from the text color can be selected to fill other text detection boxes in the target text detection box, so as to avoid the color of the filled area being the same as the text color, which affects the text recognition. Figure 18 A schematic diagram of filling other text detection boxes in the target text detection box with color in some embodiments is shown, and the text detection box L1 in the target text detection box L is filled. The text color in the target text detection box L is black, so the fill color is selected to be different from black, which can be white. By filling the text detection box L1 with white, the influence of the text detection box L1 on the target text detection box L is eliminated.

[0157] In some embodiments, when it is detected that there are other text detection boxes in the target text detection box L, the controller can perform image recognition on the target text detection box L to identify the background pattern in the target text detection box L, such as texture, etc. The controller can fill the identified background pattern into other text detection boxes to avoid the influence of other text detection boxes on the target text detection box L.

[0158] In some embodiments, after the adjustment of all the text detection boxes, the region selected by each adjusted text detection box is a text region. The controller can perform text recognition processing on the text region to obtain the text information contained in each text region, i.e., the text information contained in the user interface. Specifically, the controller can use OCR technology to perform text recognition on the text region, or can use other text recognition technology to process the text region. In the embodiments of the present application, the OCR technology is exemplarily described, but is not specifically limited.

[0159] The embodiments of the present application also provide a text recognition method applied to a display device, as shown in the method, the method comprises the following steps of: Figure 19

[0160] Step 1901, in response to a text recognition instruction input by a user, generating at least one text detection box in a user interface, the text detection box being used to detect text information in the user interface.

[0161] Step 1901, adjusting the at least one text detection box to obtain at least one text region, the text region being a region selected by the adjusted text detection box, and the text region containing a piece of text information.

[0162] Step 1901, performing text recognition processing on the at least one text region to obtain at least one piece of text information contained in the user interface.

[0163] The same and similar parts among the various embodiments in the present specification can be referred to each other, and will not be described here again.

[0164] Those skilled in the art can clearly understand that the technology in the embodiments of the present application can be realized by means of software and necessary general hardware platforms. Based on such understanding, the technical solutions in the embodiments of the present application can be embodied in the form of a software product, which can be stored in a storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the method of the various embodiments or some parts of the embodiments of the present application.

[0165] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some or all of the technical features; and such modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

[0166] ​For the sake of explanation, the foregoing descriptions have been presented in terms of specific embodiments. However, it is to be appreciated that specific embodiments described herein are not intended to limit the scope of the present application, which is defined with reference to the following claims. Various modifications and changes can be made thereto by those skilled in the art which fall within the scope of the present application as defined by the following claims. The embodiments were chosen and described in order to explain the principles of the application and the practical application and to enable others skilled in the art to understand for implementing various embodiments and with various modifications as are suited to the particular use contemplated.

Claims

1. A display device, characterized in that, include: The monitor is configured to display the user interface; The controller is configured as follows: In response to a text recognition command input by a user, at least one text detection box is generated in the user interface, the text detection box being used to detect text information in the user interface; The at least one text detection box is adjusted to obtain at least one text region, which is the area selected by the adjusted text detection box, and the text region includes a piece of text information; wherein, the adjustment of the at least one text detection box includes: obtaining a first text detection box adjacent to the text detection box based on the position information of the text detection box in the user interface; determining whether the text detection box and the first text detection box are of the same type according to a preset text detection box type condition; if they are of the same type, checking whether the length of the text detection box meets a preset length condition; if it meets the condition, not adjusting the text detection box; if it does not meet the condition, adjusting the length of the text detection box according to a preset length formula; The at least one text region is subjected to character recognition processing to obtain at least one piece of text information.

2. The display device according to claim 1, characterized in that, The controller is further configured to: In the step of generating at least one text detection box in the user interface, The user interface is captured to obtain a screenshot image; Text detection is performed on the screenshot image to obtain at least one text detection box and detection information, the detection information including the position information and size information of each text detection box in the user interface.

3. The display device according to claim 2, characterized in that, The controller is further configured to: Before performing the step of adjusting the at least one text detection box. Determine whether the size information of the text detection box meets the preset size threshold; If yes, then the step of adjusting the at least one text detection box is performed; if no, then the text detection box is deleted.

4. The display device according to claim 2, characterized in that, The controller is further configured to: In the step of adjusting the at least one text detection box... The first angle and the second angle of the text detection box are obtained based on the position information of the text detection box in the user interface. The first angle and the second angle are the angles of the two diagonals and the horizontal direction of the text detection box, respectively. The expanded size of the text detection box is determined based on the first angle and the second angle; The text detection box is expanded according to the expanded size.

5. The display device according to claim 2, characterized in that, The controller is further configured to: In the step of adjusting the at least one text detection box... The tilt angle of the text detection box is obtained based on its position information in the user interface. When the tilt angle is not detected to be a preset value, a perspective transformation is performed on the text detection box to obtain the text area corresponding to the text detection box.

6. The display device according to claim 5, characterized in that, The controller is further configured to: In the step of performing perspective transformation processing on the text detection box based on coordinate information Obtain the coordinate information of the four vertices of the text detection box; Determine the maximum and minimum values ​​of the x-coordinate and the maximum and minimum values ​​of the y-coordinate in the coordinate information; Calculate the perspective transformation matrix based on the maximum and minimum values ​​of the x-coordinate and the maximum and minimum values ​​of the y-coordinate; The text detection box is subjected to perspective transformation processing based on the perspective transformation matrix to obtain the text region.

7. The display device according to claim 2, characterized in that, The controller is further configured to: In the step of adjusting the at least one text detection box... When other text detection boxes are detected in the text detection box, all pixels in the text detection box are clustered to obtain multiple types of pixels, and each type of pixel corresponds to a color; A color is selected to fill other text detection boxes existing in the text detection box, thus obtaining the text area corresponding to the text detection box.

8. The display device according to claim 7, characterized in that, The controller is further configured to: In the step of performing clustering processing on all pixels in the text detection box A preset number of clustered pixels are set in the text detection box, and each clustered pixel corresponds to a pixel type; All pixels in the text detection box are classified according to clustered pixels, and each pixel is assigned to the pixel type corresponding to the nearest clustered pixel. The clustered pixels are iteratively updated based on the classification results, and all pixels are reclassified based on the updated clustered pixels until convergence is achieved, resulting in a preset number of pixel types. The color corresponding to each pixel type is the color of the clustered pixels at the time of convergence.

9. A text recognition method, applied to a display device, characterized in that, The method includes: In response to a text recognition command input by a user, at least one text detection box is generated in the user interface, the text detection box being used to detect text information in the user interface; The at least one text detection box is adjusted to obtain at least one text region, which is the area selected by the adjusted text detection box, and the text region includes a piece of text information; wherein, the adjustment of the at least one text detection box includes: obtaining a first text detection box adjacent to the text detection box based on the position information of the text detection box in the user interface; determining whether the text detection box and the first text detection box are of the same type according to a preset text detection box type condition; if they are of the same type, checking whether the length of the text detection box meets a preset length condition; if it meets the condition, not adjusting the text detection box; if it does not meet the condition, adjusting the length of the text detection box according to a preset length formula; The at least one text region is subjected to character recognition processing to obtain at least one piece of text information.

Images