A display device and a method for target detection thereof

Abstract

This invention discloses a display device and a method for target detection thereof, comprising: a display for displaying a user interface; and a controller configured to perform: acquiring at least one frame of an image containing various target objects; extracting features from the image to obtain a feature set for each target object; determining a detection head corresponding to each target object based on depth information of each target object in the image, wherein different detection heads are used to detect target objects of different sizes in the image; and using each detection head to detect features in the feature set to determine the position and category of each target object in the image. This method is used to detect the position and category of each target object in an image more quickly.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a display device and a method for target detection thereof. Background Technology

[0002] After years of development, artificial intelligence (AI) technology has regained attention and application in the industry. With the widespread use of cameras in smart TVs, combining AI technology with cameras has become a new trend in the smart TV industry. In modern deep learning algorithm research, the model structure of backbone network + task-specific detection head has become a standard design pattern. The backbone network is used for feature extraction, while different detection heads are used to detect and identify objects of different sizes.

[0003] Currently, when TV manufacturers perform multi-target detection on images captured by cameras, they need to use various detection heads to traverse the same target in the image and finally obtain the detection results of all detection heads. They then select the detection result with the highest confidence among the detection results of each detection head for the same target. However, in this method, each detection head needs to traverse and detect the same target in the image, which results in a large amount of computation for target detection, requires a lot of storage resources, and has a slow detection speed. Summary of the Invention

[0004] This invention provides a display device and a method for target detection thereof, which determines a detection head corresponding to each target object based on the depth information of each target object in an image, and uses the corresponding detection head to detect the features extracted from each target object in the image, thereby detecting the position and category of each target object in the image more quickly.

[0005] In a first aspect, an embodiment of the present invention provides a display device comprising:

[0006] A monitor is used to display the user interface.

[0007] The controller is configured to execute:

[0008] Acquire at least one frame of image captured, containing each target object;

[0009] Feature extraction is performed on the image to obtain feature sets for each of the target objects;

[0010] Based on the depth information of each target object in the image, a detection head corresponding to each target object is determined, wherein different detection heads are used to detect target objects of different sizes in the image;

[0011] The features in the feature set are detected using each of the detection heads to determine the location and category of each target object in the image.

[0012] The display device provided in this embodiment can select a suitable detection head for feature detection of target objects of different sizes in the image based on the depth information of each target object in the image, thereby detecting the position and category of the target object more quickly.

[0013] In some embodiments, the controller is specifically configured to perform:

[0014] Based on the depth information of each target object, the distance between each target object and the depth camera is determined;

[0015] Based on the distance between each target object and the depth camera, a detection head corresponding to each target object is determined.

[0016] This embodiment can determine the distance between each target object and the depth camera in the shooting scene based on the depth information, thereby selecting a detection head of different size for the target object at different distances and improving the speed of target detection.

[0017] In some embodiments, after acquiring at least one frame of an image containing each of the target objects, and before performing feature extraction on the image, the controller is further configured to perform:

[0018] The image is cropped to obtain a cropped image containing at least one of the target objects.

[0019] Before performing feature extraction on the image, this embodiment also obtains a cropped image containing the target object to be tested by cropping the image, thereby performing feature extraction on the cropped image, avoiding the extraction of redundant features of non-target objects, and improving the accuracy of target detection.

[0020] In some embodiments, after obtaining the cropped image containing at least one of the target objects, the controller is specifically configured to perform:

[0021] Feature extraction is performed on the cropped image to obtain feature sets for each of the target objects.

[0022] This embodiment enables feature extraction of the cropped image after image cropping, avoiding the extraction of redundant features of non-target objects, thereby improving the accuracy of target detection.

[0023] In some embodiments, if multiple frames of images containing various target objects are acquired, the controller is specifically configured to perform:

[0024] For each frame of the multi-frame image except the first frame, the cropping position of at least one of the target objects in the current frame image is determined based on the position of each target object detected in the previous frame image.

[0025] Based on the cropping position and preset size, the current frame image is cropped to obtain a cropped image containing at least one of the target objects.

[0026] This embodiment can predict the position of the target object in the current frame image based on the position of the target object determined in the previous frame image, thereby more accurately determining the cropping position for cropping and obtaining a cropped image containing the target object.

[0027] In some embodiments, the controller is specifically configured to perform:

[0028] Based on the depth information of each target object contained in the image, determine the cropping position and cropping size of each target object contained in the image;

[0029] The image is cropped according to the cropping position and the cropping size to obtain a cropped image containing at least one of the target objects.

[0030] This embodiment can also determine the cropping position and size of each target object based on its depth information in the image. It's easy to understand that the closer the target object is to the depth camera, the larger its size in the image, and thus the larger the corresponding cropping size. Conversely, the farther the target object is from the depth camera, the smaller its size in the image, and thus the smaller the corresponding cropping size. By using the depth information of the target object, the cropped image of the measured target object can be accurately cropped, improving the detection accuracy.

[0031] In some embodiments, the controller is specifically configured to perform:

[0032] Determine the product of a preset coefficient and the depth information of the target object;

[0033] The difference between the original size of the image and the product value is determined as the cropping size of the target object.

[0034] This embodiment can accurately crop the image of the target object by using the depth information of the target object, thereby improving the detection accuracy.

[0035] Secondly, an embodiment of the present invention provides a target detection method, comprising:

[0036] Acquire at least one frame of image captured, containing each target object;

[0037] Feature extraction is performed on the image to obtain feature sets for each of the target objects;

[0038] Based on the depth information of each target object in the image, a detection head corresponding to each target object is determined, wherein different detection heads are used to detect target objects of different sizes in the image;

[0039] The features in the feature set are detected using each of the detection heads to determine the location and category of each target object in the image.

[0040] In some embodiments, determining the detection head corresponding to each of the target objects based on the depth information of each target object in the image includes:

[0041] Based on the depth information of each target object in the image, the distance between each target object and the depth camera is determined;

[0042] Based on the distance between each target object and the depth camera, a detection head corresponding to each target object is determined.

[0043] In some embodiments, after acquiring at least one frame of an image containing each target object, and before performing feature extraction on the image, the method further includes:

[0044] The image is cropped to obtain a cropped image containing at least one of the target objects;

[0045] In some embodiments, after obtaining a cropped image containing at least one of the target objects, feature extraction is performed on the image to obtain a feature set for each target object, including:

[0046] Feature extraction is performed on the cropped image to obtain feature sets for each of the target objects.

[0047] In some embodiments, if multiple frames of images containing various target objects are acquired, then cropping the images to obtain a cropped image containing at least one of the target objects includes:

[0048] For each frame of the multi-frame image except the first frame, the cropping position of at least one of the target objects in the current frame image is determined based on the position of each target object detected in the previous frame image.

[0049] Based on the cropping position and preset size, the current frame image is cropped to obtain a cropped image containing at least one of the target objects.

[0050] In some embodiments, cropping the image to obtain a cropped image containing at least one of the target objects includes:

[0051] Based on the depth information of each target object contained in the image, determine the cropping position and cropping size of each target object contained in the image;

[0052] The image is cropped according to the cropping position and the cropping size to obtain a cropped image containing at least one of the target objects.

[0053] In some embodiments, determining the cropping size of each target object in the image based on the depth information of each target object contained in the image includes:

[0054] Determine the product of a preset coefficient and the depth information of the target object;

[0055] The difference between the original size of the image and the product value is determined as the cropping size of the target object.

[0056] Thirdly, embodiments of the present invention also provide a target detection apparatus, comprising:

[0057] The acquisition unit is used to acquire at least one frame of image captured, which contains each target object;

[0058] An extraction unit is used to extract features from the image to obtain feature sets for each of the target objects;

[0059] The determining unit is configured to determine a detection head corresponding to each of the target objects based on the depth information of each target object in the image, wherein different detection heads are used to detect target objects of different sizes in the image;

[0060] The detection unit is used to detect features in the feature set using each of the detection heads, and to determine the position and category of each target object in the image.

[0061] Fourthly, embodiments of the present invention also provide a computer storage medium having a computer program stored thereon, which, when executed by a processor, is used to implement the steps of the method described in the first aspect above.

[0062] These or other aspects of this application will become more apparent in the following description of embodiments. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device, provided by an embodiment of the present invention.

[0065] Figure 2 A configuration block diagram of a control device 100 provided in an embodiment of the present invention;

[0066] Figure 3 A hardware configuration block diagram of a display device 200 provided in an embodiment of the present invention;

[0067] Figure 4 This is a schematic diagram of the software configuration in a display device 200 provided in an embodiment of the present invention;

[0068] Figure 5 This is a schematic diagram of the icon control interface display of an application in a display device 200 provided in an embodiment of the present invention;

[0069] Figure 6 A schematic diagram of a display device provided in an embodiment of the present invention;

[0070] Figure 7A This is a schematic diagram of the first type of cutting provided in an embodiment of the present invention;

[0071] Figure 7B This is a schematic diagram of a second type of cutting provided in an embodiment of the present invention;

[0072] Figure 7C This is a schematic diagram of the third type of cutting provided in an embodiment of the present invention;

[0073] Figure 7D This is a schematic diagram of the fourth type of cutting provided in an embodiment of the present invention;

[0074] Figure 7E This is a schematic diagram of the fifth type of cutting provided in an embodiment of the present invention;

[0075] Figure 7F This is a schematic diagram of the sixth cutting method provided in an embodiment of the present invention;

[0076] Figure 7G This is a schematic diagram of the seventh cutting method provided in an embodiment of the present invention;

[0077] Figure 8A This is a schematic diagram of a first positional relationship provided in an embodiment of the present invention;

[0078] Figure 8B This is a schematic diagram of a second positional relationship provided in an embodiment of the present invention;

[0079] Figure 8C This is a schematic diagram of a third positional relationship provided in an embodiment of the present invention;

[0080] Figure 9 A schematic diagram of a scene for target detection using a display device according to an embodiment of the present invention;

[0081] Figure 10 A detailed flowchart of a target detection method is provided for an embodiment of the present invention;

[0082] Figure 11 This is a flowchart illustrating an implementation of a target detection method according to an embodiment of the present invention;

[0083] Figure 12 A schematic diagram of a target detection device provided in an embodiment of the present invention;

[0084] Figure 13 This is a schematic diagram of a target detection device provided in an embodiment of the present invention. Detailed Implementation

[0085] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0086] In this embodiment of the invention, the term "and / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0087] The application scenarios described in the embodiments of this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. Those skilled in the art will understand that with the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems. In the description of this invention, unless otherwise stated, "multiple" means two or more.

[0088] After years of development, artificial intelligence (AI) technology has regained attention and application in the industry. With the widespread use of cameras in display devices, combining AI technology with cameras has become a new trend in the display device industry. In modern deep learning algorithm research, the backbone network + task-specific detection head model structure has become a standard design pattern. The backbone network is used for feature extraction; in this embodiment, it is used to extract features of various target objects in the captured image, while different detection heads are used to detect and identify objects of different sizes.

[0089] It should be noted that the backbone network in this embodiment specifically refers to the convolutional layer structure used for feature extraction in the deep learning model, including but not limited to MobileNet and ResNet, which are mainly used for feature extraction from the input image. The detection head in this embodiment specifically refers to the convolutional layers used for classification detection in the deep learning model, mainly including convolutional layers of different sizes and depthwise separable convolutional layers of different sizes. The structural connections of each convolutional layer and depthwise separable convolutional layer in the detection head can be specifically designed according to the specific target object being detected; this embodiment does not impose excessive limitations on this. The detection head is mainly used to detect the location and category of each target object from the extracted features. The detection head can be understood as a detection box (anchor box) that traverses and detects the image. The detection result obtained by the detection head can be understood as the detection box with the highest confidence determined from each detection box, and the location and category of the target object contained in the detection box are determined based on the location of the detection box and the category labeled on the detection box.

[0090] Currently, when TV manufacturers perform multi-target detection on images captured by cameras, they need to use each detection head to traverse each target in the image and finally obtain the detection results of all detection heads. The detection result with the highest confidence is then selected. If the target to be detected in the image is a car or a pedestrian, and multiple different detection heads (detection boxes) are used for target detection at the same time, then in the specific detection process, the image is divided into 3×3 image blocks on average. For each image block, multiple detection heads are used for detection. Each image block will output multiple detection boxes obtained by multiple detection heads. Detection boxes with low probability values ​​are discarded. For each target, the NMS algorithm is used to obtain the final prediction box. The position and category of the target are determined based on the prediction box.

[0091] It's easy to understand that current object detection methods require each detection head to detect every single object in the image. This results in a large computational load, consuming significant storage resources and leading to slow detection speeds. To address this slow speed, the core idea of ​​this embodiment is to utilize the depth information of each target object within the image captured by a depth camera. This allows for the selection of the most suitable detection head for each target object, enabling the detection of target objects of different sizes within the image using different detection heads. This results in faster and more targeted detection of the location and category of each target object in the image, effectively improving detection speed.

[0092] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device according to one or more embodiments of this application, such as... Figure 1 As shown, a user can operate the display device 200 via a mobile terminal 300 and a control device 100. The control device 100 can be a remote control, and communication between the remote control and the display device includes infrared protocol communication, Bluetooth protocol communication, wireless or other wired methods to control the display device 200. The user can input user commands through buttons on the remote control, voice input, control panel input, etc., to control the display device 200. In some embodiments, a mobile terminal, tablet computer, computer, laptop computer, and other smart devices can also be used to control the display device 200.

[0093] In some embodiments, the mobile terminal 300 can install software applications with the display device 200 to achieve connection and communication via network communication protocols, enabling one-to-one control operations and data communication. Audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 for synchronous display. The display device 200 also communicates with the server 400 via various communication methods. The display device 200 can communicate via a local area network (LAN), wireless local area network (WLAN), and other networks. The server 400 can provide various content and interactive features to the display device 200. The display device 200 can be a liquid crystal display, an OLED display, or a projection display device. In addition to providing broadcast television reception functions, the display device 200 can also be equipped with a smart network television function that provides computer support.

[0094] Figure 2 An exemplary block diagram of the configuration of the control device 100 according to an exemplary embodiment is shown. Figure 2As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 can receive user input commands and convert them into commands that the display device 200 can recognize and respond to, acting as an intermediary for interaction between the user and the display device 200. The communication interface 130 is used for external communication and includes at least one of a Wi-Fi chip, a Bluetooth module, NFC, or a replacement module. The user input / output interface 140 includes at least one of a microphone, a touchpad, a sensor, buttons, or a replacement module.

[0095] Figure 3 A hardware configuration block diagram of a display device 200 according to an exemplary embodiment is shown. For example... Figure 3 The display device 200 shown includes at least one of the following: a tuner / demodulator 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface 280. The controller includes a central processing unit, a video processor, an audio processor, a graphics processor, RAM, ROM, and a first to nth interface for input / output. The display 260 can be at least one of a liquid crystal display, an OLED display, a touch display, and a projection display, and can also be a projection device and a projection screen. The tuner / demodulator 210 receives broadcast television signals via wired or wireless reception and demodulates audio and video signals, such as EPG data signals, from multiple wireless or wired broadcast television signals. The detector 230 is used to collect signals from the external environment or signals interacting with the external environment. The controller 250 and the tuner / demodulator 210 can be located in different separate devices; that is, the tuner / demodulator 210 can also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0096] In some embodiments, the controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in memory. The controller 250 controls the overall operation of the display device 200. The user can input user commands through a graphical user interface (GUI) displayed on the monitor 260, and the user input interface receives the user input commands through the GUI. Alternatively, the user can input user commands by inputting specific sounds or gestures, and the user input interface receives the user input commands by recognizing the sounds or gestures through sensors.

[0097] In some embodiments, a "user interface" is the medium through which an application or operating system interacts and exchanges information with a user, enabling the conversion between the internal form of information and a form acceptable to the user. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be an icon, window, control, or other interface element displayed on the screen of an electronic device. Controls can include at least one of the visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0098] Figure 4 This is a schematic diagram of the software configuration in a display device 200 according to one or more embodiments of this application, such as... Figure 4 As shown, the system is divided into four layers, from top to bottom: the Applications layer (referred to as the "Application Layer"), the Application Framework layer (referred to as the "Framework Layer"), the Android runtime and system library layer (referred to as the "System Runtime Layer"), and the kernel layer. The kernel layer contains at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, Wi-Fi driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0099] Figure 5 This is a schematic diagram showing the icon control interface of an application in a display device 200 according to one or more embodiments of this application, such as... Figure 5 As shown, the application layer contains at least one application whose corresponding icon control can be displayed on the screen, such as: live TV application icon control, video-on-demand application icon control, media center application icon control, application center icon control, game application icon control, etc. Live TV applications can provide live television from different signal sources. Video-on-demand applications can provide video from different storage sources. Unlike live TV applications, video-on-demand provides video display from certain storage sources. Media center applications can provide applications for playing various multimedia content. The application center can provide storage for various applications.

[0100] like Figure 6 As shown, this embodiment provides a display device, including:

[0101] Display 600 is used to display the user interface;

[0102] Controller 601 is configured to execute:

[0103] Acquire at least one frame of image containing each of the target objects; extract features from the image to obtain a feature set for each target object; determine a detection head corresponding to each target object based on the depth information of each target object, wherein different detection heads are used to detect target objects of different sizes in the image; use each detection head to detect features in the feature set to determine the position and category of each target object in the image.

[0104] It should be noted that the display device in this embodiment uses images captured by a depth camera and depth information of various target objects in the images to extract features from the images. Then, it uses a detection head matched with each target object to detect the extracted features, thereby detecting the location and category of the corresponding target object in a targeted, accurate, and fast manner. The location of the target object includes its size in the image.

[0105] In some examples, the display devices involved in this embodiment include, but are not limited to, smart TVs, smart tablets, ultra-large LED screens, and video wall displays. Currently, most display devices have high resolution. When performing multi-target detection on high-resolution images, since the size of each target is unknown, it is necessary to use multiple detection heads to detect the same target object, which makes the target detection extremely costly and unsuitable for target detection on high-resolution display devices. In this application, the depth information of each target object obtained by the depth camera is used to select the detection head corresponding to each target object. Since different depth information can represent the distance between the target object and the depth camera in the actual scene, the closer the distance, the larger the proportion of the captured image on the display screen of the display device. However, the larger the proportion, the lower the resolution. Different detection heads are suitable for different resolutions. In other words, different detection heads are suitable for target objects of different sizes. After obtaining the depth information of the target objects in the image, based on the size of each target object in the image (the proportion it occupies on the screen), this implementation can use different detection heads to detect the target objects separately based on this characteristic. It is not necessary for each detection head to detect the same target object. Thus, different detection heads can be used to detect different target objects, improving the target detection speed. This can be better applied to display devices and improve the target detection speed of display devices.

[0106] It should be noted that in this embodiment, when performing target detection on a single target object, since a corresponding detection head is selected for the target object based on its depth information in the image, it is not necessary to use different detection heads to detect the target object. Only one corresponding detection head is needed. Furthermore, because this detection head is determined based on the target object's depth information, the selected detection head is suitable for that target object. Accurate detection of the target object can be achieved using only one detection head, effectively improving the efficiency of target detection. If target detection is performed on multiple target objects, based on the same principle, only the detection head corresponding to each target object needs to be used to accurately detect each target.

[0107] In some examples, the display device in this embodiment may have a built-in depth camera integrated into the display device, or it may be connected to an external depth camera. This embodiment does not impose too many limitations on this. The depth camera involved in this embodiment can capture images containing various target objects and can obtain the depth information of each target object contained in the image.

[0108] The difference between the depth camera in this embodiment and a regular camera is that, in addition to acquiring planar images, it can also obtain the depth information of the target object being photographed, that is, the three-dimensional position and size information of the target object in the image. The depth information includes, but is not limited to, three-dimensional position and three-dimensional size. This allows the controller to acquire the environmental conditions of each target object in the captured image and the three-dimensional data (three-dimensional position and three-dimensional size) of the target object. By determining the three-dimensional position of the target object, the distance between the target object and the depth camera in the actual shooting environment is determined, thereby selecting a matching detection head for each target object in the image for detection.

[0109] The technology used by the depth camera in this embodiment includes, but is not limited to, any of the following:

[0110] 1) Time of Flight (TOF) technology;

[0111] The Time-of-Flight (TOF) principle utilizes the sensor of a depth camera to emit modulated near-infrared light, which is reflected upon encountering a target object. By calculating the time difference between the emission and reflection of the light, or the phase difference between the emission and reflection of the light, the distance between the target object and the depth camera can be determined.

[0112] 2) Structured Light technology;

[0113] Structured light technology projects an encoded grating or line light source onto the target object and demodulates the three-dimensional data of the target object based on the distortion produced by the grating or line light source.

[0114] 3) Binocular vision technology;

[0115] Binocular vision technology uses two ordinary cameras to calculate the distance between the target object and the depth camera by means of parallax.

[0116] In this embodiment, different depth information is obtained by using different depth cameras. However, regardless of the type of depth information obtained, the distance between the target object and the depth camera can be determined based on the obtained depth information.

[0117] This embodiment can continuously capture multiple frames of images containing at least one target object, and use the corresponding detection head to detect the target object to obtain its location and category. Alternatively, it can capture a single frame of an image containing at least one target object, and use the corresponding detection head to detect the target object to obtain its location and category. This embodiment can simultaneously detect multiple target objects in an image, using detection heads corresponding to each target object to improve the speed of target detection. It can also detect a single target object in an image using the detection head corresponding to that target object; this embodiment does not impose excessive limitations on this approach.

[0118] In some embodiments, this embodiment can determine the corresponding detection head based on the depth information of the target object. The specific implementation steps are as follows:

[0119] Step 1) Determine the distance between each target object and the depth camera based on the depth information of each target object;

[0120] In practice, if the depth information includes three-dimensional position and three-dimensional dimensions, the distance between each target object and the depth camera is determined based on the three-dimensional position and three-dimensional dimensions of each target object; or the distance between each target object and the depth camera is directly determined based on the distance information in the depth information. The specific method of determining the distance based on the depth information is determined by the technology used by the depth camera, and this embodiment does not impose excessive limitations on this.

[0121] Step 2) Determine the detection head corresponding to each target object based on the distance between each target object and the depth camera.

[0122] In this embodiment, different detection heads are used to detect target objects of different sizes in the image. The design concept of determining the detection head based on distance is that a larger target object is selected for detection of a target object that is closer to the depth camera, and a smaller target object is selected for detection of a target object that is farther from the depth camera. As the distance between the target object and the depth camera decreases, the size of the target object detected by the corresponding detection head increases. It is easy to understand that if the target object is closer to the depth camera in the actual shooting scene, it means that the target object is closer to the depth camera, and obviously the size of the captured target object in the image is larger than that of a target object that is farther away. Conversely, if the target object is farther from the depth camera in the actual shooting scene, it means that the size of the captured target object in the image is smaller than that of a target object that is farther away. For the same frame of image, the size and resolution of the target object in the image are different. If the target object is larger, the resolution is lower, and it is detected by a detection head used to detect larger objects. If the target object is smaller, the resolution is higher, and it is detected by a detection head used to detect smaller objects. This embodiment avoids using all detection heads to traverse the image by setting different detection heads for target objects of different sizes, thereby improving the detection speed.

[0123] In some embodiments, to improve the accuracy of detection, this embodiment also provides a method for cropping an image, which is used to crop the image into an image containing the target object to be tested before feature extraction, thereby avoiding feature extraction of unnecessary image information such as background, thereby extracting features about the target object to be tested as accurately as possible, detecting the target object, and thus improving the accuracy of detection.

[0124] In some embodiments, after acquiring at least one frame of an image containing each of the target objects, and before performing feature extraction on the image, the processor is further configured to perform:

[0125] The image is cropped to obtain a cropped image containing at least one of the target objects.

[0126] In practice, the image is cropped to include the target object being tested. The target object can be one or more, and this embodiment does not impose excessive limitations on this. It should be noted that if there are multiple target objects, after determining the position of each target object in the image, the image is cropped based on the position of each target object, ensuring that all target objects are included.

[0127] In some embodiments, in conjunction with the above-described cropping method, this embodiment also provides a target detection method to improve detection speed and accuracy. The core idea of ​​this method is: after extracting features from the acquired image containing each target object, the image is cropped to obtain an image containing the target object to be tested, and other non-target objects and irrelevant background features are cropped out. Thus, when performing feature extraction, more feature information of the target object to be tested can be obtained, while interference features of non-target objects to be tested are removed. Therefore, in the subsequent classification and detection process of the extracted features, the target object to be tested can be identified and detected more accurately.

[0128] In this embodiment, after acquiring at least one frame of image containing each of the target objects, each frame is cropped. The purpose of cropping is to obtain a cropped image containing at least one of the target objects, where the at least one target object can be understood as the target object of interest to the user. After cropping, feature extraction is performed on the cropped image to obtain a feature set of each of the target objects in the cropped image. Based on the depth information of each of the target objects in the cropped image, a detection head corresponding to each of the target objects is determined, wherein different detection heads are used to detect target objects of different sizes in the cropped image. Thus, each detection head is used to perform targeted detection on each feature in the feature set to determine the position and category of each of the target objects in the cropped image.

[0129] In some embodiments, this embodiment provides any of the following cutting methods, specifically including:

[0130] Method 1) Determine the cropped image by using the position of the target object detected in the previous frame.

[0131] If multiple frames of images containing various target objects are acquired, then for each frame of the multiple frames excluding the first frame, the cropped image is determined by cropping in the following manner:

[0132] 1-1. Based on the positions of each of the target objects detected in the previous frame image, determine the cropping position of at least one of the target objects in the current frame image;

[0133] 1-2. Based on the cropping position and preset size, crop the current frame image to obtain a cropped image containing at least one of the target objects.

[0134] It should be noted that in this method, the first frame image is not cropped; only subsequent frames are cropped. If there is only one target object, the current frame image is cropped according to the cropping position of the target object and a preset size to obtain a cropped image containing the target object. If there are multiple target objects, the current frame image is cropped according to the cropping position of each target object and a preset size, where the preset size ensures that each target object is included, to obtain a cropped image containing all target objects. In this cropping method, the cropping size and shape are preset in this embodiment. The cropping position of the target object is used to represent the center coordinates of the area formed by the target object in the image. Based on the preset size (cropping size), the current frame image is cropped according to the preset size, with the cropping position of the target object in the previous frame image as the center.

[0135] like Figure 7A As shown, if the detected target object in the previous frame is located at the top left corner, the cropping position (center coordinates) of the target object is taken as the center of the region. According to the preset region size, the image is cropped into region 1 with the center of the region as the center and the region size as the preset size. Then, the cropped image of region 1 is used to extract features and detect the target object in region 1. Since only the features of region 1 are extracted and the features of the remaining regions are removed, the target detection can be performed more accurately in the subsequent feature detection process.

[0136] like Figure 7B As shown, if the detected target object in the previous frame is located at the upper right corner, the cropping position (center coordinates) of the target object is taken as the center of the region. According to the preset region size, the image is cropped into region 2 with the center of this region as the center and the region size is the preset size. Then, the cropped image of region 2 is used to extract features and detect the target object in region 2. Since only the features of region 2 are extracted and the features of the remaining regions are removed, the target detection can be performed more accurately in the subsequent feature detection process.

[0137] like Figure 7C As shown, if the target object detected in the previous frame is located on the bottom, the cropping position (center coordinates) of the target object is taken as the center of the region. The image is cropped into region 3 with the center of the region as the center and the size of the region as the preset size. Then, the cropped image of region 3 is used to extract features and target detection is performed on the target object in region 3. Since only the features of region 3 are extracted and the features of the remaining regions are removed, the target detection can be performed more accurately in the subsequent feature detection process.

[0138] like Figure 7D As shown, if the target object detected in the previous frame is located on the top, the cropping position (center coordinates) of the target object is taken as the center of the region. The image is cropped into region 4 with the center of the region as the center and the size of the region as the preset size. Then, the cropped image of region 4 is used to extract features and target detection is performed on the target object in region 4. Since only the features of region 4 are extracted and the features of the remaining regions are removed, the target detection can be performed more accurately in the subsequent feature detection process.

[0139] like Figure 7E As shown, if the detected target object in the previous frame is located at the lower left corner, the cropping position (center coordinates) of the target object is taken as the center of the region. The image is cropped into region 5 with the center of the region as the center and the size of the region as the preset size. Then, the cropped image of region 5 is used to extract features and detect the target object in region 5. Since only the features of region 5 are extracted and the features of the remaining regions are removed, the target detection can be performed more accurately in the subsequent feature detection process.

[0140] like Figure 7F As shown, if the target object detected in the previous frame is located at the lower right corner, the cropping position (center coordinates) of the target object is taken as the center of the region. The image is cropped into region 6 with the center of the region as the center and the size of the region as the preset size, according to the preset size of the region. Then, the cropped image of region 4 is used to extract features and target detection is performed on the target object in region 4. Since only the features of region 4 are extracted and the features of the remaining regions are removed, the target detection can be performed more accurately in the subsequent feature detection process.

[0141] like Figure 7G As shown, if the detected target object in the previous frame is located in the middle, the cropping position (center coordinates) of the target object is taken as the center of the region. According to the preset region size, the image is cropped into region 7 with the center of the region as the center and the region size is the preset size. Then, the cropped image of region 4 is used to extract features and target detection is performed on the target object in region 4. Since only the features of region 4 are extracted and the features of the remaining regions are removed, the target detection can be performed more accurately in the subsequent feature detection process.

[0142] Method 2) Use depth information to determine the cropped image.

[0143] It should be noted that this method can determine the position and size of each target object in each frame of an image containing each target object using depth information, and thus determine the cropped image through the following steps:

[0144] 2-1. Based on the depth information of each target object contained in the image, determine the cropping position and cropping size of each target object contained in the image;

[0145] In practice, the depth information of each target object includes, but is not limited to, the three-dimensional position and three-dimensional size of each target object in the image. The cropping position of each target object is determined based on its three-dimensional position, and the cropping size of each target object is determined based on its three-dimensional size.

[0146] In some embodiments, the cutting dimensions can also be determined in the following ways:

[0147] Determine the product of the preset coefficient lamda and the depth information distance of the target object;

[0148] The difference between the original size of the image and the product value is determined as the cropping size of the target object; wherein, the cropping size can be determined by the following formula (1):

[0149] ObjectWidth=CropWidth-lamda*distance formula (1);

[0150] Where lambda represents a preset coefficient, and lambda is greater than zero; ObjectWidth represents the cropping size of the target object, CropWidth represents the original size of the image, and distance is used to represent depth information.

[0151] 2-2. Based on the cropping position and the cropping size, crop the image to obtain a cropped image containing at least one of the target objects.

[0152] In practice, the image is cropped according to the cropping position and size of the target object being measured, to determine a cropped image containing the target object. The target object can be one or more, without further limitation here.

[0153] In some embodiments, this embodiment uses the human body and the human body's hand as target objects, and provides a method for human body and gesture detection. The core idea of ​​this method is to use a depth camera to capture at least one frame of image containing the human body and the human body's hand, and to crop the acquired image to obtain a cropped image containing only the human body and the human body's hand, so as to remove non-human body and hand feature information, thereby extracting the features of the human body and hand in the image, and obtaining a feature set of the human body and hand; and to determine the detection head corresponding to the human body and hand respectively based on the depth information of the human body and hand in the cropped image; thereby using the detection head corresponding to the human body to detect human body-related features in the feature set, the position of the human body in the image is determined, and using the detection head corresponding to the hand to detect hand-related features in the feature set, the position of the hand in the image is determined, thereby effectively improving the detection speed.

[0154] It should be noted that the actual positional relationship between the captured human body and hands, and the depth camera, includes, but is not limited to, any of the following:

[0155] Type 1, such as Figure 8A As shown, the distance between the hand and the depth camera is closer than the distance between the human body and the depth camera. Therefore, based on the different distances between the hand and the human body and the depth camera, the corresponding detection head is determined. Moreover, the detection head for the hand is larger than the detection head for the human body, so the hand and the human body can be detected more quickly by using different detection heads.

[0156] The second type, such as Figure 8B As shown, the distance between the hand and the depth camera is closer than the distance between the human body and the depth camera. Therefore, the hand and the human body are considered to be on the same plane with respect to the depth camera. The detection head for the human body is determined based on the distance from the human body to the depth camera, and the detection head for the human body is used to detect the hand. In other words, the same detection head is used to detect both the human body and the hand.

[0157] The third type, such as Figure 8C As shown, the distance between the hand and the depth camera is greater than the distance between the human body and the depth camera. Therefore, based on the different distances between the hand and the human body and the depth camera, corresponding detection heads are determined respectively. Moreover, the detection head for the hand is smaller than the detection head for the human body, so the hand and the human body can be detected more quickly by using different detection heads.

[0158] In some embodiments, such as Figure 9 As shown in the figure, this embodiment provides a scene diagram of target detection using a display device, including a display device 900 and a depth camera 901, wherein:

[0159] The depth camera is positioned in the center of the display device, and its shooting range is within a 60° angle from the depth camera. In this scenario, because the human body is moving, the resolution of the human body and its gestures displayed on the display device varies depending on the distance between the human body and the display device. For example, when the human body is 1 meter away from the display device, the depth camera can capture a range of 1.15 meters; when the human body is 2 meters away, the depth camera can capture a range of 2.31 meters; when the human body is 3 meters away, the depth camera can capture a range of 3.47 meters; when the human body is 3.5 meters away, the depth camera can capture a range of 4.04 meters, and so on. In this scenario, target detection is typically performed on family members, including family members and their gestures. The appropriate detection head varies depending on the distance between the moving human body and the display device. For example, when detecting only the human body, if the distance between the human body and the depth camera is 1 meter, the human body occupies a relatively large proportion of the screen, so a detection head suitable for low resolution can be selected. If the distance is 3 meters, the human body occupies a relatively small proportion of the screen, so a detection head suitable for high resolution can be selected. Similarly, for multi-target detection, only the detection head corresponding to each target object is used, effectively improving detection speed.

[0160] In some embodiments, such as Figure 10 As shown in the figure, this embodiment provides a detailed method flow for target detection, as detailed below:

[0161] Step 1000: Acquire at least one frame of image containing each of the target objects;

[0162] Step 1001: Determine the cropping position and cropping size of each target object in the image based on the depth information of each target object contained in the image;

[0163] Step 1002: Crop the image according to the cropping position and the cropping size to obtain a cropped image containing at least one of the target objects;

[0164] Step 1003: Extract features from the cropped image to obtain a feature set of at least one target object;

[0165] Step 1004: Determine the distance between at least one target object and the depth camera based on the depth information of at least one target object;

[0166] Step 1005: Determine the detection head corresponding to at least one target object based on the distance between at least one target object and the depth camera;

[0167] Step 1006: Use each of the detection heads to detect the features in the feature set, and determine the position and category of each of the target objects in the image.

[0168] Based on the same inventive concept, this embodiment of the invention also provides a method for target detection. Since this method is the same method used by the display device in this embodiment of the invention, and the principle of solving the problem by this method is similar to that of the display device, the implementation of this method can be referred to the implementation of the display device, and repeated details will not be described again.

[0169] like Figure 11 As shown, the implementation process of this method is as follows:

[0170] Step 1100: Acquire at least one frame of image containing each target object;

[0171] Step 1101: Extract features from the image to obtain feature sets for each of the target objects;

[0172] Step 1102: Based on the depth information of each target object in the image, determine the detection head corresponding to each target object, wherein different detection heads are used to detect target objects of different sizes in the image;

[0173] Step 1103: Use each of the detection heads to detect the features in the feature set, and determine the position and category of each target object in the image.

[0174] In some embodiments, determining the detection head corresponding to each of the target objects based on the depth information of each target object in the image includes:

[0175] Based on the depth information of each target object in the image, the distance between each target object and the depth camera is determined;

[0176] Based on the distance between each target object and the depth camera, a detection head corresponding to each target object is determined.

[0177] In some embodiments, after acquiring at least one frame of an image containing each target object, and before performing feature extraction on the image, the method further includes:

[0178] The image is cropped to obtain a cropped image containing at least one of the target objects;

[0179] In some embodiments, after obtaining a cropped image containing at least one of the target objects, feature extraction is performed on the image to obtain a feature set for each target object, including:

[0180] Feature extraction is performed on the cropped image to obtain feature sets for each of the target objects.

[0181] In some embodiments, if multiple frames of images containing various target objects are acquired, then cropping the images to obtain a cropped image containing at least one of the target objects includes:

[0182] For each frame of the multi-frame image except the first frame, the cropping position of at least one of the target objects in the current frame image is determined based on the position of each target object detected in the previous frame image.

[0183] Based on the cropping position and preset size, the current frame image is cropped to obtain a cropped image containing at least one of the target objects.

[0184] In some embodiments, cropping the image to obtain a cropped image containing at least one of the target objects includes:

[0185] Based on the depth information of each target object contained in the image, determine the cropping position and cropping size of each target object contained in the image;

[0186] The image is cropped according to the cropping position and the cropping size to obtain a cropped image containing at least one of the target objects.

[0187] In some embodiments, determining the cropping size of each target object in the image based on the depth information of each target object contained in the image includes:

[0188] Determine the product of a preset coefficient and the depth information of the target object;

[0189] The difference between the original size of the image and the product value is determined as the cropping size of the target object.

[0190] Based on the same inventive concept, this embodiment of the invention also provides a target detection device. Since this device is the same as the device in the method of this embodiment of the invention, and the principle of the device in solving the problem is similar to that of the method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0191] like Figure 12As shown, the device includes a processor 1200 and a memory 1201. The memory 1201 stores programs executable by the processor 1200. The processor 1200 reads the programs from the memory 1201 and performs the following steps:

[0192] Acquire at least one frame of image containing each of the target objects;

[0193] Feature extraction is performed on the image to obtain feature sets for each of the target objects;

[0194] Based on the depth information of each target object in the image, a detection head corresponding to each target object is determined, wherein different detection heads are used to detect target objects of different sizes in the image;

[0195] The features in the feature set are detected using each of the detection heads to determine the location and category of each target object in the image.

[0196] In some embodiments, the processor 1200 is specifically configured to perform:

[0197] Based on the depth information of each target object in the image, the distance between each target object and the depth camera is determined;

[0198] Based on the distance between each target object and the depth camera, a detection head corresponding to each target object is determined.

[0199] In some embodiments, after acquiring at least one frame of an image containing each of the target objects, and before performing feature extraction on the image, the processor 1200 is further configured to perform:

[0200] The image is cropped to obtain a cropped image containing at least one of the target objects.

[0201] In some embodiments, after obtaining the cropped image containing at least one of the target objects, the processor 1200 is specifically configured to perform:

[0202] Feature extraction is performed on the cropped image to obtain feature sets for each of the target objects.

[0203] In some embodiments, if multiple frames of images containing various target objects are acquired, the processor 1200 is specifically configured to execute:

[0204] For each frame of the multi-frame image except the first frame, the cropping position of at least one of the target objects in the current frame image is determined based on the position of each target object detected in the previous frame image.

[0205] Based on the cropping position and preset size, the current frame image is cropped to obtain a cropped image containing at least one of the target objects.

[0206] In some embodiments, the processor 1200 is specifically configured to perform:

[0207] Based on the depth information of each target object contained in the image, determine the cropping position and cropping size of each target object contained in the image;

[0208] The image is cropped according to the cropping position and the cropping size to obtain a cropped image containing at least one of the target objects.

[0209] In some embodiments, the processor 1200 is specifically configured to perform:

[0210] Determine the product of a preset coefficient and the depth information of the target object;

[0211] The difference between the original size of the image and the product value is determined as the cropping size of the target object.

[0212] Based on the same inventive concept, this embodiment of the invention also provides a target detection device. Since this device is the same as the device in the method of this embodiment of the invention, and the principle of the device in solving the problem is similar to that of the method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0213] like Figure 13 As shown, the device includes:

[0214] The acquisition unit 1300 is used to acquire at least one frame of image containing each target object;

[0215] Extraction unit 1301 is used to extract features from the image to obtain feature sets of each target object;

[0216] The determining unit 1302 is used to determine a detection head corresponding to each of the target objects based on the depth information of each target object in the image, wherein different detection heads are used to detect target objects of different sizes in the image;

[0217] The detection unit 1303 is used to detect features in the feature set using each of the detection heads, and to determine the position and category of each target object in the image.

[0218] In some embodiments, the present invention also provides a computer storage medium storing a computer program that, when executed by a processor, performs the following steps:

[0219] Acquire at least one frame of image containing each of the target objects;

[0220] Feature extraction is performed on the image to obtain feature sets for each of the target objects;

[0221] Based on the depth information of each target object in the image, a detection head corresponding to each target object is determined, wherein different detection heads are used to detect target objects of different sizes in the image;

[0222] The features in the feature set are detected using each of the detection heads to determine the location and category of each target object in the image.

[0223] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0224] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 Devices that specify the functions in one or more boxes.

[0225] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction device, which is implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0226] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0227] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A display device, characterized in that, include: A monitor is used to display the user interface. The controller is configured to execute: Acquire at least one frame of image captured, containing each target object; Feature extraction is performed on the image to obtain feature sets for each of the target objects; Based on the depth information of each target object in the image, the distance between each target object and the depth camera is determined; Based on the distance between each target object and the depth camera, a detection head corresponding to each target object is determined, wherein different detection heads are used to detect target objects of different sizes in the image; the detection head is a convolutional layer in a deep learning model used for classification and detection; and each target object has its own corresponding detection head; The features in the feature set are detected using each of the detection heads to determine the location and category of each target object in the image.

2. The display device according to claim 1, characterized in that, After acquiring at least one frame of image containing each of the target objects, and before performing feature extraction on the image, the controller is further configured to perform: The image is cropped to obtain a cropped image containing at least one of the target objects.

3. The display device according to claim 2, characterized in that, After obtaining the cropped image containing at least one of the target objects, the controller is specifically configured to execute: Feature extraction is performed on the cropped image to obtain feature sets for each of the target objects.

4. The display device according to any one of claims 1 to 3, characterized in that, If multiple frames of images containing various target objects are acquired, the controller is specifically configured to execute: For each frame of the multi-frame image except the first frame, the cropping position of at least one of the target objects in the current frame image is determined based on the position of each target object detected in the previous frame image. Based on the cropping position and preset size, the current frame image is cropped to obtain a cropped image containing at least one of the target objects.

5. The display device according to any one of claims 1 to 3, characterized in that, The controller is specifically configured to execute: Based on the depth information of each target object contained in the image, determine the cropping position and cropping size of each target object contained in the image; The image is cropped according to the cropping position and the cropping size to obtain a cropped image containing at least one of the target objects.

6. The display device according to claim 5, characterized in that, The controller is specifically configured to execute: Determine the product of a preset coefficient and the depth information of the target object; The difference between the original size of the image and the product value is determined as the cropping size of the target object.

7. A method for target detection, characterized in that, The method includes: Acquire at least one frame of image captured, containing each target object; Feature extraction is performed on the image to obtain feature sets for each of the target objects; Based on the depth information of each target object in the image, the distance between each target object and the depth camera is determined; Based on the distance between each target object and the depth camera, a detection head corresponding to each target object is determined, wherein different detection heads are used to detect target objects of different sizes in the image; the detection head is a convolutional layer in a deep learning model used for classification and detection; and each target object has its own corresponding detection head; The features in the feature set are detected using each of the detection heads to determine the location and category of each target object in the image.

8. The method according to claim 7, characterized in that, After acquiring at least one frame of image containing each target object, and before performing feature extraction on the image, the method further includes: The image is cropped to obtain a cropped image containing at least one of the target objects.

9. The method according to claim 8, characterized in that, After obtaining a cropped image containing at least one of the target objects, feature extraction is performed on the image to obtain a feature set for each target object, including: Feature extraction is performed on the cropped image to obtain feature sets for each of the target objects.

Images