Image detection and recognition device and detection and recognition method thereof

Abstract

The embodiment of the present application discloses an image detection and recognition device and a detection and recognition method thereof, which are used to optimize the detection and recognition of the display area of ​​a television screen, realize real-time screen color picking, thereby controlling the color change of the light strip and improving the user experience. The method of the embodiment of the present application includes: a priori information storage module, a camera module, an image processing module, and a communication module; wherein: the priori information storage module is used to save the priori information required by the image detection and recognition device when it is first enabled, and transmit the priori information to the image processing module when the image detection and recognition device is first enabled; the camera module is used to obtain image data of a rectangular display area and transmit the image data to the image processing module; the image processing module is used to obtain the position information of the rectangular display area through an inter-frame relative difference algorithm; the communication module is used for information interaction between the image detection and recognition device and a host or terminal.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of IoT smart home, and in particular to an image detection and recognition device and a detection and recognition method thereof. Background Art

[0002] With the popularization of IoT home smart devices, more and more smart application scenarios have emerged. Immersive smart TV light strip is a product that integrates smart lighting and smart cameras. It mainly uses the camera to obtain the color of the TV screen in real time, and then controls the light strip installed on the back of the TV to display the corresponding color, realizing an immersive viewing experience.

[0003] The main technologies in this type of product can be summarized into three parts, namely TV screen area recognition, real-time screen color picking, and real-time light strip control. TV screen area recognition is currently mainly achieved through manual calibration of products on the market, that is, through the visual screen, users manually complete the area marking through interface interaction.

[0004] However, this calibration process will lead to complicated product operation and redundant steps during human interaction, poor user experience and inaccurate calibration results. Summary of the invention

[0005] The embodiments of the present application provide an image detection and recognition device and a detection and recognition method thereof, which have higher accuracy, thereby optimizing the detection and recognition of the display area of ​​the TV screen, realizing real-time screen color picking, and are easy to operate, thereby improving the user experience.

[0006] An image detection and recognition device provided in an embodiment of the present application is applied to a home smart device, comprising: a priori information storage module, a camera module, an image processing module, and a communication module; wherein:

[0007] A priori information storage module, used to store the priori information required by the image detection and recognition device when it is first activated, and transmit the priori information to the image processing module when the image detection and recognition device is first activated, so that the image processing module determines the position information of the rectangular display area according to the priori information;

[0008] A camera module, used for acquiring image data of a rectangular display area, and transmitting the image data to the image processing module, so that the image processing module determines position information of the rectangular display area according to the image data;

[0009] An image processing module, used for obtaining the position information of the rectangular display area by using an inter-frame relative difference algorithm;

[0010] The communication module is used for information exchange between the image detection and recognition device and the host or terminal, and is connected to the host or terminal by wired connection or wireless connection, so that various hosts or terminals can connect to the image detection and recognition device and have image detection and recognition functions.

[0011] Optionally, the camera module acquires image data of the rectangular display area through a fisheye camera installed at a midpoint position close to any side of the rectangular display area.

[0012] Optionally, the prior information is acquired by manual calibration, and the prior information is stored in the prior information storage module.

[0013] Optionally, the prior information includes the coordinates of the first vertex, the second vertex, the first midpoint and the area where the relative edge is located on the installation edge; wherein the installation edge is the edge of the rectangular display area close to the installation of the fisheye camera, the relative edge is the edge of the rectangular display area opposite to the installation edge, and the first midpoint is the midpoint between the first vertex and the second vertex of the installation edge.

[0014] An image detection and recognition method provided in an embodiment of the present application is applied to any of the above-mentioned image detection and recognition devices, including:

[0015] Acquire prior information, the prior information including the coordinates of a first vertex, a second vertex, a first midpoint, and an area where an opposite side is located on the installation side; wherein the installation side is an edge of the rectangular display area close to where the fisheye camera is installed, the opposite side is an edge of the rectangular display area opposite to the installation side, and the first midpoint is a midpoint between the first vertex and the second vertex of the installation side;

[0016] Acquire first image data and second image data of a rectangular display area, wherein the first image data and the second image data are image data of the rectangular display area under different brightness states;

[0017] Applying an inter-frame relative difference algorithm to the first image data and the second image data to determine the third vertex coordinates, the fourth vertex coordinates and the second midpoint coordinates of the opposite side of the rectangular display area, wherein the second midpoint is the midpoint of the rectangular display area located at the opposite side;

[0018] In combination with the prior data, the third midpoint coordinates are determined by the first vertex coordinates and the third vertex coordinates, and the fourth midpoint coordinates are determined by the second vertex coordinates and the fourth vertex coordinates; wherein the third midpoint coordinates are the coordinates of the midpoint between the first vertex and the third vertex, and the fourth midpoint coordinates are the coordinates of the midpoint between the second vertex and the fourth vertex;

[0019] The first vertex coordinates, the second vertex coordinates, the third vertex coordinates, the fourth vertex coordinates, the first midpoint coordinates, the second midpoint coordinates, the third midpoint coordinates, and the fourth midpoint coordinates are determined as position information of the rectangular display area.

[0020] Optionally, the obtaining of prior information includes:

[0021] Acquire a plurality of sample image data, wherein the sample image data is obtained by photographing rectangular display areas of different sizes by the fisheye camera;

[0022] Establish a target coordinate system based on the sample image, with the upper left corner vertex of the sample image as the coordinate origin, the horizontal rightward direction along the upper left corner vertex of the image as the positive direction of the X axis, and the vertical downward direction along the upper left corner vertex of the image as the positive direction of the Y axis;

[0023] Marking the first vertex coordinates, the second vertex coordinates, the first midpoint coordinates and the area where the opposite edge is located in each sample image on the target coordinate system by a manual calibration method;

[0024] The first vertex coordinates, the second vertex coordinates, the first midpoint coordinates and the area where the relative edge is located are determined according to the annotation results of the multiple sample images and used as prior information.

[0025] Optionally, acquiring the first image data and the second image data of the rectangular display area includes:

[0026] The first image data and the second image data of the rectangular display area in different brightness states are obtained by interacting with the user terminal.

[0027] Optionally, the applying an inter-frame relative difference algorithm to the first image data and the second image data to determine the third vertex coordinates, the fourth vertex coordinates, and the second midpoint coordinates of the opposite sides of the rectangular display area comprises:

[0028] According to the relative pixel changes of the first image data and the second image data under different brightness states, marking the area where the pixels become larger as a white pixel area, and marking the area where the pixels become smaller as a black pixel area, to obtain a differential image;

[0029] Marking the area where the relative edge is located on the differential image according to the prior information; wherein the area where the relative edge is located is represented by a rectangular frame, and the rectangular frame includes a part of the white pixel area and a part of the black pixel area;

[0030] Taking the top left corner vertex of the rectangular frame as the starting point and one pixel as the basic unit, count the number of pixels in a horizontal row in the rectangular frame each time, and calculate the first ratio of the number of white pixels to the total number of pixels in a row;

[0031] Traversing the rectangular frame along the Y-axis direction, when the first ratio reaches a preset first condition, confirming that the ordinate of the current row is the ordinate of the third vertex and the fourth vertex of the opposite side, and the abscissa of the third vertex and the fourth vertex is the abscissa of the two sides of the rectangular frame perpendicular to the X-axis, and obtaining the coordinates of the third vertex and the fourth vertex;

[0032] Continue to traverse the rectangular frame along the Y-axis direction. When the first ratio reaches the preset second condition, confirm that the vertical coordinate of the current row is the vertical coordinate of the second midpoint located on the opposite side, take the middle value of the horizontal coordinate of the third vertex and the horizontal coordinate of the fourth vertex as the horizontal coordinate of the second midpoint, and obtain the coordinates of the second midpoint.

[0033] An image detection and recognition device provided in an embodiment of the present application is characterized by comprising:

[0034] CPU, memory and input / output interface;

[0035] The memory is a short-term storage memory or a persistent storage memory;

[0036] The central processing unit is configured to communicate with the memory and execute instruction operations in the memory to perform the aforementioned image detection and recognition method.

[0037] A computer-readable storage medium provided in an embodiment of the present application includes instructions, and when the instructions are executed on a computer, the computer executes the aforementioned image detection and recognition method.

[0038] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0039] The image data of the rectangular display area is obtained through the camera module, and the position information of the rectangular display area is obtained by the image processing module through the relative difference algorithm between frames and combined with the manually calibrated prior information, which has higher accuracy, thereby optimizing the detection and recognition of the display area of ​​the TV screen and realizing real-time screen color picking. It is easy to operate and improves the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A diagram showing the structure of an image detection and recognition device provided in an embodiment of the present application;

[0041] Figure 2 A schematic diagram of an implementation of the image detection and recognition method provided in an embodiment of the present application;

[0042] Figure 3 A schematic diagram of the structure of an image detection and recognition device provided in an embodiment of the present application;

[0043] Figure 4 The interactive interface of the user terminal provided in the embodiment of the present application;

[0044] Figure 5 Screenshots of the monitoring screen of the fisheye camera provided in the embodiment of the present application and screenshots of the interactive interface of the corresponding user terminal;

[0045] Figure 6 The interactive interface of the user terminal when the television provided in the embodiment of the present application is turned on;

[0046] Figure 7 The interactive interface of the user terminal when the television is turned off provided in the embodiment of the present application;

[0047] Figure 8 A screenshot of a television set provided in an embodiment of the present application displayed on a user terminal when the television set is turned off;

[0048] Fig. 9 A screenshot of a television provided in an embodiment of the present application displayed on a user terminal when it is turned on;

[0049] Fig.10 A differential image provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The embodiments of the present application provide an image detection and recognition device and a detection and recognition method thereof, which have higher accuracy, thereby optimizing the detection and recognition of the display area of ​​the TV screen, realizing real-time screen color picking, and are easy to operate, thereby improving the user experience.

[0051] With the popularization of IoT home smart devices, more and more smart application scenarios have emerged. Immersive smart TV light strip is a product that integrates smart lighting and smart cameras. It mainly uses the camera to obtain the color of the TV screen in real time, and then controls the light strip installed on the back of the TV to display the corresponding color, realizing an immersive viewing experience.

[0052] The main technologies in this type of product can be summarized into three parts, namely TV screen area recognition, real-time screen color picking, and real-time light strip control. TV screen area recognition is currently mainly achieved through manual calibration of products on the market, that is, through the visual screen, users manually complete the area marking through interface interaction.

[0053] However, this calibration process will lead to complicated product operation and redundant steps during human interaction, poor user experience and inaccurate calibration results.

[0054] Based on this, the present application embodiment provides an image detection and recognition device, which is applied to home smart devices. Figure 1An implementation of the image detection and recognition device provided in the embodiment of the present application includes: a priori information storage module 101, a camera module 102, an image processing module 103 and a communication module 104. Among them:

[0055] The prior information storage module 101 is used to store the prior information required by the image detection and recognition device when it is first activated, and transmit the prior information to the image processing module when the image detection and recognition device is first activated, so that the image processing module determines the position information of the rectangular display area according to the prior information.

[0056] In combination with the usage scenario of the present device, when the present device is used to detect a television or other device with a display function, detection, identification and calibration of the display area should be performed when it is first activated after installation. It is worth noting that the rectangular display area in the embodiment of the present application is used to refer to the rectangular display area of ​​other devices with a display function such as a television screen. The initial activation in the embodiment of the present application is not limited to the first time the device is turned on and used, but should also include the situation where the device performs a system reset.

[0057] Prior information is some information or parameters used to enable the device to perform its functions, obtained by the technical developers of the device through multiple experiments during the device design and manufacturing stage. It is pre-stored in the prior information storage module 101 during the device production stage to achieve the technical effect of the device.

[0058] Specifically, in order to achieve the technical effect of detecting and identifying the position information of the rectangular display area, the device acquires images of TV screens of multiple sizes in advance during the design and development stage. Specifically, by selecting TV screens covering 35-70 inches as samples, it is verified through experiments that when the camera of the device is installed in a specified area (such as the top edge or the bottom edge of the TV screen close to the midpoint), the two vertices of the installation edge of the camera always fall within a certain range of the monitoring screen image seen by the camera, so the specific coordinate positions of these two vertices and the midpoint of the installation edge are acquired in this way and set as fixed coordinate values ​​to be pre-stored in the prior information storage module 101. In addition, during the development and test stage, the developers also found that the bottom edge of the TV screen can still fall within a fixed range after being deformed by the camera imaging, and this range can be represented by a rectangular frame, so the position of this rectangular frame is also saved as prior information in the prior information storage module 101.

[0059] The camera module 102 is used to obtain image data of the rectangular display area and transmit the image data to the image processing module, so that the image processing module determines the position information of the rectangular display area according to the image data.

[0060] The image processing module 103 is used to obtain the position information of the rectangular display area by using an inter-frame relative difference algorithm.

[0061] Multiple frames of images under different brightness states are captured by the camera module 102, and a frame difference algorithm is used to obtain relative changes between different frames for the multiple frames. Since the pixels contained in the multiple frames of images acquired in this embodiment include both the TV screen part and the environment part where the TV is placed, when the TV screen brightness is adjusted, there is a relative change trend in the pixel changes at the same position of each frame of the image. In this embodiment, the position information of the rectangular display area is obtained through this relative change trend. The method adopted is improved based on the relative change trend relative to the frame difference algorithm known in the industry, so the description method of "frame relative difference algorithm" is adopted in this application to reflect the specific implementation process.

[0062] The communication module 104 is used for information exchange between the image detection and recognition device and the host or terminal: it is provided with one or more interfaces including USB, serial port SPI bus, connected to the host or terminal by wired connection or wireless connection, so that various hosts or terminals with different interfaces can be connected to the image detection and recognition device and have image detection and recognition functions.

[0063] In this embodiment, image data of the rectangular display area is acquired through the camera module 102, and the image processing module 103 is used to acquire the position information of the rectangular display area through the relative difference algorithm between frames and combined with manually calibrated prior information, which has higher accuracy, thereby optimizing the detection and recognition of the display area of ​​the TV screen and realizing real-time screen color capture. It is easy to operate and improves the user experience.

[0064] Another implementation of the image detection and recognition device provided in the embodiment of the present application includes: a priori information storage module 101, a camera module 102, an image processing module 103 and a communication module 104. Among them:

[0065] The prior information storage module 101 is used to store the prior information required by the image detection and recognition device when it is first activated, and transmit the prior information to the image processing module when the image detection and recognition device is first activated, so that the image processing module determines the position information of the rectangular display area according to the prior information.

[0066] In combination with the usage scenario of the present device, when the present device is used to detect a television or other device with a display function, detection, identification and calibration of the display area should be performed when it is first activated after installation. It is worth noting that the rectangular display area in the embodiment of the present application is used to refer to the rectangular display area of ​​other devices with a display function such as a television screen. The initial activation in the embodiment of the present application is not limited to the first time the device is turned on and used, but should also include the situation where the device performs a system reset.

[0067] Prior information is some information or parameters used to enable the device to perform its functions, obtained by the technical developers of the device through multiple experiments during the device design and manufacturing stage. It is pre-stored in the prior information storage module 101 during the device production stage to achieve the technical effect of the device.

[0068] Specifically, in order to achieve the technical effect of detecting and identifying the position information of the rectangular display area, the device acquires images of TV screens of multiple sizes in advance during the design and development stage. Specifically, by selecting TV screens covering 35-70 inches as samples, it is verified through experiments that when the camera of the device is installed in a specified area (such as the top edge or the bottom edge of the TV screen close to the midpoint), the two vertices of the installation edge of the camera always fall within a certain range of the monitoring screen image seen by the camera, so the specific coordinate positions of these two vertices and the midpoint of the installation edge are acquired in this way and set as fixed coordinate values ​​to be pre-stored in the prior information storage module 101. In addition, during the development and test stage, the developers also found that the bottom edge of the TV screen can still fall within a fixed range after being deformed by the camera imaging, and this range can be represented by a rectangular frame, so the position of this rectangular frame is also saved as prior information in the prior information storage module 101.

[0069] The camera module 102 is used to obtain image data of the rectangular display area and transmit the image data to the image processing module, so that the image processing module determines the position information of the rectangular display area according to the image data.

[0070] The camera module 102 in the device uses a fisheye camera to obtain a 180° wide-angle monitoring picture, so that when the device is installed on the top or bottom of a TV screen, the picture of the TV screen can still be relatively completely reflected in the monitoring picture. The camera module 102 can also have other wide-angle cameras that can achieve the same technical effect, which is not limited here.

[0071] When installing the camera, the user can interact through the user terminal so that the device can obtain specific installation location information to facilitate the subsequent steps of detecting the TV display area. The specific user terminal interaction interface can be as follows Figure 4shown.

[0072] For example, when a fisheye camera is installed above a TV screen, the surveillance screen screenshots taken by the camera and the corresponding installation method are displayed in the user terminal interface screenshots as shown in the figure below. Figure 5 shown.

[0073] As above Figure 5 This is a screenshot of the surveillance screen obtained by installing the fisheye camera near the midpoint of the top edge of a 70-inch TV. The difference between the two methods is that the vertex positions of the TV are different. In the specific calculation, the vertex positions can be transformed by rotation. In the process of manually calibrating to obtain prior information, a coordinate system is first established through the above screenshots, where the vertex in the upper left corner of the image is the origin (0, 0) in the computer coordinate system, and the horizontal direction to the right along the vertex in the upper left corner of the image is the positive direction of the X axis in the computer coordinate system, that is, the horizontal coordinate. Vertically downward along the upper left corner of the image is the positive direction of the Y axis in the computer coordinate system, that is, the vertical coordinate.

[0074] The specific steps are as follows:

[0075] After installing the camera on top of TVs of different sizes according to the product usage rules, obtain a screenshot of the monitoring screen;

[0076] First determine the coordinates of the top midpoint of the fisheye image, such as Figure 5 The actual TV top midpoint marked in;

[0077] Then get the coordinates of the two vertices on the top and left of the TV, such as Figure 5 The upper left and upper right corners are marked;

[0078] The third step is to obtain the three coordinate points of the bottom left, center and right of the TV, such as Figure 5 The lower left and lower right corners are marked. The black frame in the figure can determine the upper and lower ranges and length ranges of the bottom edge of the TV. After calibration and measurement, it can be confirmed that the bottom edges of different 35-70-inch TVs can fall within the black frame;

[0079] The midpoint of the left and right borders of the TV is calculated from the midpoint of the line connecting the upper and lower vertices on the left and right sides, such as Figure 5 The actual left and right midpoints of the TV are marked in;

[0080] After the above calibration steps, the position of the TV in the camera image can be preliminarily determined, providing support for subsequent algorithms to detect more accurate vertex coordinates.

[0081] The image processing module 103 is used to obtain the position information of the rectangular display area by using an inter-frame relative difference algorithm.

[0082] Multiple frames of images under different brightness states are captured by the camera module 102, and a frame difference algorithm is used to obtain relative changes between different frames for the multiple frames. Since the pixels contained in the multiple frames of images acquired in this embodiment include both the TV screen part and the environment part where the TV is placed, when the TV screen brightness is adjusted, there is a relative change trend in the pixel changes at the same position of each frame of the image. In this embodiment, the position information of the rectangular display area is obtained through this relative change trend. The method adopted is improved based on the relative change trend relative to the frame difference algorithm known in the industry, so the description method of "frame relative difference algorithm" is adopted in this application to reflect the specific implementation process.

[0083] In this embodiment, two frames of images of the TV in the on and off states can be obtained by the user interacting with the user terminal to provide image data for subsequent algorithms. The interactive interface is as follows Figure 6 , Figure 7 shown.

[0084] The following screenshots show the two frames of images of the TV in the off and on states displayed on the user terminal: Figure 8 , Fig. 9 As shown, the two pictures are inter-frame differentiated to obtain the differential image as follows Fig.10 .

[0085] In the differential process, the relative change trend that the brightness of the TV screen area will increase and the brightness of the background area will decrease when the TV is turned from off to on is used to mark the area where the pixels in the image become larger as white areas, and vice versa as black areas. The final differential image is as follows: Fig.10 As shown. The large white area in the image is the TV screen area, and the other black areas are the background area outside the TV. The approximate area of ​​the TV can be quickly located by combining prior information. The left and right vertices and the midpoint of the top of the TV in this embodiment are manually calibrated data, and then combined with the relative difference algorithm between frames, the bottom vertex of the TV is located.

[0086] The communication module 104 is used for information exchange between the image detection and recognition device and the host or terminal, and is connected to the host or terminal by wired connection or wireless connection, so that various hosts or terminals can be connected to the image detection and recognition device and have image detection and recognition functions.

[0087] In this embodiment, the camera module 102 obtains image data of the rectangular display area through a fisheye camera, so that the TV screen can appear completely in the monitoring screen, and the image processing module 103 obtains the position information of the rectangular display area through the relative difference algorithm between frames and combined with the manually calibrated prior information, which has higher accuracy, thereby optimizing the detection and recognition of the display area of ​​the TV screen, realizing real-time screen color selection, and specifically embodies the implementation process of user terminal interaction. It can be seen that it is easy to operate and can improve the user experience.

[0088] See also Figure 2 An implementation of the image detection and recognition method provided in an embodiment of the present application includes steps 201 to 205.

[0089] 201. Obtain prior information.

[0090] The prior information includes the coordinates of the first vertex, the second vertex, the first midpoint and the area where the relative edge is located on the installation edge; wherein the installation edge is the edge of the rectangular display area close to the installation fisheye camera, the relative edge is the edge of the rectangular display area opposite to the installation edge, and the first midpoint is the midpoint between the first vertex and the second vertex of the installation edge.

[0091] Specifically, multiple sets of sample image data are obtained, where the sample image data is a monitoring screen obtained by shooting rectangular display areas of different sizes by a fisheye camera. A target coordinate system is established based on the sample image, with the upper left corner vertex of the sample image as the coordinate origin, the horizontal rightward direction along the upper left corner vertex of the image is the positive direction of the X axis, and the vertical downward direction along the upper left corner vertex of the image is the positive direction of the Y axis.

[0092] In the manual calibration process, the midpoint of the top edge of the sample image is first determined as the first midpoint, and the coordinates of the first midpoint are determined according to its specific position in the target coordinate system. Secondly, multiple sample images are enlarged one by one, and the developer visually observes the enlarged sample images and determines the specific positions of the first vertex and the second vertex of the installation edge in each sample image and marks them, and determines the first vertex coordinates and the second vertex coordinates in the multiple sample images in combination with the target coordinate system. The first vertex coordinates in the multiple sample images are formed into a first vertex coordinate set, and the second vertex coordinates in the multiple sample images are formed into a second vertex coordinate set. The first vertex coordinate set is averaged, and the average coordinates are determined as the first vertex coordinates obtained in the prior information through manual calibration. Similarly, this method can also be used to obtain the second vertex coordinates in the prior information.

[0093] During the manual calibration process, the developers also found that the bottom edge of the TV screen can still fall within a fixed range after being deformed by the camera imaging, and this range can be represented by a rectangular frame, so the position of this rectangular frame is also used as prior information.

[0094] 202. Acquire first image data and second image data of a rectangular display area.

[0095] The first image data and the second image data are image data of the rectangular display area at different brightness states. Specifically, the fisheye camera can be controlled to obtain the first image data and the second image data of the rectangular display area at different brightness states in the form of interaction with the user terminal.

[0096] 203. Apply an inter-frame relative difference algorithm to the first image data and the second image data to determine the third vertex coordinates, the fourth vertex coordinates, and the second midpoint coordinates of the opposite sides of the rectangular display area.

[0097] According to the relative pixel changes of the first image data and the second image data under different brightness states, the area where the pixels become larger is marked as a white pixel area, and the area where the pixels become smaller is marked as a black pixel area, and a differential image is obtained. The area where the relative edge is located is marked on the differential image according to the prior information; wherein the area where the relative edge is located is represented by a rectangular frame, and the rectangular frame includes part of the white pixel area and part of the black pixel area. Taking the upper left corner vertex of the rectangular frame as the starting point and a pixel point as the basic unit, the number of pixels in a row in the horizontal direction of the rectangular frame is counted each time, and the first proportion of the number of white pixels in the total number of pixels in a row is calculated. The rectangular frame is traversed along the Y-axis direction. When the first proportion reaches the preset first condition, it is confirmed that the ordinate of the current row is the ordinate of the third vertex and the fourth vertex of the relative edge, and the abscissa of the third vertex and the fourth vertex is the abscissa of the two sides of the rectangular frame perpendicular to the X-axis, and the coordinates of the third vertex and the fourth vertex are obtained. Continue to traverse the rectangular frame along the Y-axis direction. When the first ratio reaches the preset second condition, confirm that the vertical coordinate of the current row is the vertical coordinate of the second midpoint located on the opposite side, take the middle value of the horizontal coordinate of the third vertex and the horizontal coordinate of the fourth vertex as the horizontal coordinate of the second midpoint, and obtain the second midpoint coordinate.

[0098] Specifically, the first preset condition may be: during the traversal process, when the proportion of white pixels is less than 90% twice in a row. The second preset condition may be: during the traversal process, when the proportion of white pixels is less than 60% twice in a row. The above data is the optimal value of the effect obtained by the developer during the development and testing process. In the actual application process, the fault tolerance of this solution can be enhanced by adjusting the threshold to obtain a relatively better detection and recognition result, which is not limited here.

[0099] 204. In combination with the priori data, determine the third midpoint coordinates by using the first vertex coordinates and the third vertex coordinates, and determine the fourth midpoint coordinates by using the second vertex coordinates and the fourth vertex coordinates.

[0100] The third midpoint coordinates are the coordinates of the midpoint between the first vertex and the third vertex, and the fourth midpoint coordinates are the coordinates of the midpoint between the second vertex and the fourth vertex.

[0101] 205. Determine the first vertex coordinates, the second vertex coordinates, the third vertex coordinates, the fourth vertex coordinates, the first midpoint coordinates, the second midpoint coordinates, the third midpoint coordinates, and the fourth midpoint coordinates as position information of the rectangular display area.

[0102] In this embodiment, image data of the rectangular display area is acquired through a fisheye camera so that the TV screen can appear completely in the monitoring screen. The position information of the rectangular display area is acquired with higher accuracy through an inter-frame relative difference algorithm combined with manually calibrated prior information, thereby optimizing the detection and recognition of the TV screen display area, realizing real-time screen color selection, and improving the user experience.

[0103] Figure 3 300 is a schematic diagram of the structure of an image detection and recognition device provided in an embodiment of the present application. The image detection and recognition device 300 may include one or more central processing units (CPU) 301 and a memory 305. The memory 305 stores one or more application programs or data.

[0104] The memory 305 may be a volatile storage or a persistent storage. The program stored in the memory 305 may include one or more modules, each of which may include a series of instruction operations in the image detection and recognition device. Furthermore, the central processing unit 301 may be configured to communicate with the memory 305, and execute a series of instruction operations in the memory 305 on the image detection and recognition device 300.

[0105] The image detection and recognition device 300 may also include one or more power supplies 302, one or more wired or wireless network interfaces 303, one or more input and output interfaces 304, and / or one or more operating systems, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0106] The CPU 301 can execute the aforementioned Figure 1 to Figure 2 The operations performed by the image detection and recognition device in the illustrated embodiment will not be described in detail here.

[0107] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0108] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0109] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0110] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0111] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

Claims

1. An image detection and recognition device, applied to home smart devices, characterized in that: include: Prior information storage module, camera module, image processing module, communication module; wherein: A priori information storage module, used to store the priori information required by the image detection and recognition device when it is first activated, and transmit the priori information to the image processing module when the image detection and recognition device is first activated, so that the image processing module determines the position information of the rectangular display area according to the priori information; A camera module, used for acquiring image data of a rectangular display area, and transmitting the image data to the image processing module, so that the image processing module determines position information of the rectangular display area according to the image data; An image processing module, used to determine the position information of the rectangular display area by using an inter-frame relative difference algorithm, and then determine the final position information of the rectangular display area according to the prior information and the position information of the rectangular display area determined by the inter-frame relative difference algorithm; The communication module is used for information exchange between the image detection and recognition device and the host or terminal, and is connected to the host or terminal by wired connection or wireless connection, so that various hosts or terminals can connect to the image detection and recognition device and have image detection and recognition functions.

2. The image detection and recognition device according to claim 1, characterized in that: The camera module acquires image data of the rectangular display area through a fisheye camera installed at a midpoint position close to any side of the rectangular display area.

3. The image detection and recognition device according to claim 2, characterized in that: The prior information is obtained by manual calibration, and the prior information is stored in the prior information storage module.

4. The image detection and recognition device according to claim 3, characterized in that: The prior information includes the coordinates of the first vertex, the second vertex, the first midpoint and the area where the relative edge is located on the installation edge; wherein the installation edge is the edge of the rectangular display area close to the installation of the fisheye camera, the relative edge is the edge of the rectangular display area opposite to the installation edge, and the first midpoint is the midpoint between the first vertex and the second vertex of the installation edge.

5. An image detection and recognition method, applied to the image detection and recognition device according to any one of claims 2 to 4, characterized in that: include: Acquire prior information, the prior information including the coordinates of a first vertex, a second vertex, a first midpoint, and an area where an opposite side is located on the installation side; wherein the installation side is an edge of the rectangular display area close to where the fisheye camera is installed, the opposite side is an edge of the rectangular display area opposite to the installation side, and the first midpoint is a midpoint between the first vertex and the second vertex of the installation side; Acquire first image data and second image data of a rectangular display area, wherein the first image data and the second image data are image data of the rectangular display area under different brightness states; Applying an inter-frame relative difference algorithm to the first image data and the second image data to determine the third vertex coordinates, the fourth vertex coordinates and the second midpoint coordinates of the opposite side of the rectangular display area, wherein the second midpoint is the midpoint of the rectangular display area located at the opposite side; In combination with the prior information, the third midpoint coordinates are determined by the first vertex coordinates and the third vertex coordinates, and the fourth midpoint coordinates are determined by the second vertex coordinates and the fourth vertex coordinates; wherein the third midpoint coordinates are the coordinates of the midpoint between the first vertex and the third vertex, and the fourth midpoint coordinates are the coordinates of the midpoint between the second vertex and the fourth vertex; The first vertex coordinates, the second vertex coordinates, the third vertex coordinates, the fourth vertex coordinates, the first midpoint coordinates, the second midpoint coordinates, the third midpoint coordinates, and the fourth midpoint coordinates are determined as position information of the rectangular display area.

6. The image detection and recognition method according to claim 5, characterized in that: The obtaining of prior information comprises: Acquire a plurality of sample image data, wherein the sample image data is obtained by photographing rectangular display areas of different sizes by the fisheye camera; Establish a target coordinate system based on the sample image, with the upper left corner vertex of the sample image as the coordinate origin, the horizontal rightward direction along the upper left corner vertex of the image as the positive direction of the X axis, and the vertical downward direction along the upper left corner vertex of the image as the positive direction of the Y axis; Marking the first vertex coordinates, the second vertex coordinates, the first midpoint coordinates and the area where the opposite edge is located in each sample image on the target coordinate system by a manual calibration method; The first vertex coordinates, the second vertex coordinates, the first midpoint coordinates and the area where the relative edge is located are determined according to the annotation results of the multiple sample images and used as prior information.

7. The image detection and recognition method according to claim 6, characterized in that: The acquiring of the first image data and the second image data of the rectangular display area comprises: The first image data and the second image data of the rectangular display area in different brightness states are obtained by interacting with the user terminal.

8. The image detection and recognition method according to claim 7, characterized in that: The applying the inter-frame relative difference algorithm to the first image data and the second image data to determine the third vertex coordinates, the fourth vertex coordinates and the second midpoint coordinates of the opposite sides of the rectangular display area comprises: According to the relative pixel changes of the first image data and the second image data under different brightness states, marking the area where the pixels become larger as a white pixel area, and marking the area where the pixels become smaller as a black pixel area, to obtain a differential image; Marking the area where the relative edge is located on the differential image according to the prior information; wherein the area where the relative edge is located is represented by a rectangular frame, and the rectangular frame includes a part of the white pixel area and a part of the black pixel area; Taking the top left corner vertex of the rectangular frame as the starting point and one pixel as the basic unit, count the number of pixels in a horizontal row in the rectangular frame each time, and calculate the first ratio of the number of white pixels to the total number of pixels in a row; Traversing the rectangular frame along the Y-axis direction, when the first ratio reaches a preset first condition, confirming that the ordinate of the current row is the ordinate of the third vertex and the fourth vertex of the opposite side, and the abscissa of the third vertex and the fourth vertex is the abscissa of the two sides of the rectangular frame perpendicular to the X-axis, and obtaining the coordinates of the third vertex and the fourth vertex; Continue to traverse the rectangular frame along the Y-axis direction. When the first ratio reaches the preset second condition, confirm that the vertical coordinate of the current row is the vertical coordinate of the second midpoint located on the opposite side, take the middle value of the horizontal coordinate of the third vertex and the horizontal coordinate of the fourth vertex as the horizontal coordinate of the second midpoint, and obtain the coordinates of the second midpoint.

9. An image detection and recognition device, characterized in that: include: CPU, memory and input / output interface; The memory is a short-term storage memory or a persistent storage memory; The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the method according to any one of claims 5 to 8.

10. A computer-readable storage medium comprising instructions, which, when executed on a computer, enable the computer to perform the method according to any one of claims 5 to 8.

Images