Target detection method and display equipment

Abstract

The invention discloses a target detection method and display equipment, and the method comprises the steps: carrying out the target detection of a to-be-detected image through a target detection model, and obtaining a preliminary detection result; for first target detection data of which the confidence coefficient is greater than a first preset value and less than a second preset value, acquiring a context region image of a target frame contained in the first target detection data, and generating a target detection prompt word based on a target type contained in the first target detection data; based on the context region image and the target detection prompt word, determining the validity of the first target detection data by using a multi-modal large model; and under the condition that the first target detection data is valid, constructing a final detection result corresponding to the to-be-detected image based on the target detection data of which the confidence coefficient is greater than a second preset value and the first target detection data.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of visual detection, and in particular to a target detection method and a display device. BACKGROUND

[0002] Visual detection has a wide range of applications in various industries. Multi-modal large models can process different modal data such as text, images, videos, and audio, and realize cross-modal understanding and generation. They have stronger feature expression ability and comprehensive understanding ability, can handle more complex tasks, and can adapt to various scenarios. Multi-modal large models have shown excellent performance in the field of visual detection. However, due to the large number of model parameters and the huge amount of calculation, the response speed of multi-modal large models is slow. SUMMARY

[0003] The present application provides a target detection method and a display device to solve the problem of slow target detection speed.

[0004] In a first aspect, some embodiments provide a target detection method, comprising:

[0005] performing target detection on the to-be-detected image by using a target detection model to obtain a preliminary detection result, the preliminary detection result comprising at least one set of target detection data, the target detection data comprising a target frame, a target type, and a confidence level;

[0006] for first target detection data with a confidence level greater than a first preset value and less than a second preset value, obtaining a context region image of the target frame contained in the first target detection data, generating a target detection prompt word based on the target type contained in the first target detection data; based on the context region image and the target detection prompt word, determining the validity of the first target detection data by using a multi-modal large model;

[0007] in the case where the first target detection data is valid, constructing a final detection result corresponding to the to-be-detected image based on target detection data with a confidence level greater than the second preset value and the first target detection data;

[0008] in the case where the first target detection data is invalid, taking target detection data with a confidence level greater than the second preset value as the final detection result corresponding to the to-be-detected image.

[0009] In some embodiments, the target detection model comprises a backbone network, a neck network, and a prediction head; and the performing target detection on the to-be-detected image by using a target detection model to obtain a preliminary detection result comprises:

[0010] inputting the to-be-detected image into the backbone network, performing image feature extraction on the to-be-detected image by the backbone network to obtain a feature map;

[0011] inputting the feature map into the neck network, performing feature enhancement on the feature map by the neck network, and obtaining an enhanced feature map;

[0012] inputting the enhanced feature map into the prediction head, and outputting a preliminary detection result by the prediction head.

[0013] In some embodiments, the context region image of the target bounding box contained in the first target detection data comprises:

[0014] obtaining a non-maximum suppression-free prediction box corresponding to the first target detection data;

[0015] obtaining a maximum circumscribed rectangle box of the target bounding box contained in the first target detection data and the prediction box;

[0016] increasing the length and width of the maximum circumscribed rectangle box by a preset proportion to obtain an input box, and the region in the input box on the to-be-detected image being the context region image.

[0017] In some embodiments, the target detection prompt word is generated based on the target type contained in the first target detection data, comprising:

[0018] adding the target type contained in the first target detection data into a pre-set instruction template to obtain the target detection prompt word.

[0019] In some embodiments, the multi-modal large model comprises: a modal encoder, a modal mapper, a large language model, and a modal decoder, and the effectiveness of the first target detection data is determined based on the context region image and the target detection prompt word by using the multi-modal large model, comprising:

[0020] inputting the context region image into the modal encoder to obtain a visual feature vector, and inputting the target detection prompt word into the modal encoder to obtain a text feature vector;

[0021] inputting the visual feature vector into the modal mapper, and mapping the visual feature vector into a text modal vector by the modal mapper;

[0022] cutting the text feature vector and the text modal vector into feature slices, inputting the feature slices into the large language model, extracting target information by the large language model, inputting the target information into the decoder, and determining the effectiveness of the first target detection data based on a decoding result output by the decoder.

[0023] In some embodiments, the method further comprises:

[0024] In the target detection model training process, for a positive sample, when a confidence output by a target detection model is less than a preset threshold, the positive sample is subjected to data enhancement to expand the number of samples.

[0025] In some embodiments, the method further comprises:

[0026] obtaining a scene picture of a preset scene;

[0027] performing target extraction and region extraction on the scene picture;

[0028] inputting the scene picture, the extracted target and the extracted region into the multi-modal large model respectively for preliminary description to obtain target description information, region description information and global description information;

[0029] inputting the scene picture, the target description information, the region description information, the global description information and detailed description prompt words into the multi-modal large model for detailed description to obtain detailed description information;

[0030] constructing multi-type task data, each type of task data including related images, task instructions and answers;

[0031] based on the detailed description information and the multi-type task data, fine-tuning the multi-modal large model.

[0032] In a second aspect, some embodiments further provide a display device comprising a display and a processor:

[0033] The processor is configured to: use a target detection model to perform target detection on a to-be-detected image to obtain a preliminary detection result, the preliminary detection result comprising at least one set of target detection data, the target detection data comprising a target frame, a target type and a confidence;

[0034] for first target detection data with a confidence greater than a first preset value and less than a second preset value, obtaining a context region image of the target frame contained in the first target detection data, generating target detection prompt words based on the target type contained in the first target detection data; based on the context region image and the target detection prompt words, using a multi-modal large model to determine the effectiveness of the first target detection data;

[0035] in the case where the first target detection data is effective, based on target detection data with a confidence greater than the second preset value and the first target detection data, constructing a final detection result corresponding to the to-be-detected image;

[0036] in the case where the first target detection data is ineffective, using target detection data with a confidence greater than the second preset value as the final detection result corresponding to the to-be-detected image.

[0037] The display is configured to display a final detection result corresponding to the image to be detected.

[0038] In a third aspect, a computer-readable storage medium is provided, and the computer-readable storage medium stores a computer program. The computer program is executed by a processor to implement the target detection method.

[0039] In a fourth aspect, a computer program product is provided, and the computer program product comprises a computer program. The computer program is executed by a processor to implement the target detection method.

[0040] The target detection method, the display device, the storage medium, and the program product described above first use a target detection model to perform preliminary target detection. For suspicious targets, a multi-modal large model is used for further confirmation. This avoids the problem of slow response caused by the large size of model parameters when directly using a multi-modal large model for detection. Moreover, for suspicious targets, effective re-trial is performed based on context region images. Compared with re-trial based on the entire image to be detected, the re-trial result is more accurate. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application. Those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0042] Figure 1 The display device and the control device provided by some embodiments of the present application provide a schematic diagram of the operation scenario between the display device and the control device;

[0043] Figure 2 The display device provided by some embodiments of the present application provides a schematic diagram of the hardware configuration of the display device;

[0044] Figure 3 The control device provided by some embodiments of the present application provides a schematic diagram of the hardware configuration of the control device;

[0045] Figure 4 The display device provided by some embodiments of the present application provides a schematic diagram of the software configuration of the display device;

[0046] Figure 5 The target detection method provided by some embodiments of the present application provides a flowchart of the target detection method Figure 1 ;

[0047] Figure 6 The target detection method provided by some embodiments of the present application provides a flowchart of the target detection method Figure 2 ;

[0048] Figure 7 A diagram of a maximum circumscribed rectangle frame provided for some embodiments of the present application;

[0049] Figure 8 A diagram of detailed description information provided for some embodiments of the present application. DETAILED DESCRIPTION

[0050] The embodiments will be described in detail with reference to the drawings, of which examples are shown. In the following description, references are made to the accompanying drawings which form a part hereof, and in which are shown, by way of illustration, specific embodiments. DETAILED DESCRIPTION

[0051] It should be noted that the brief description of the terms in the present application is only for the convenience of understanding the following described embodiments, and is not intended to limit the embodiments of the present application. Unless otherwise specified, these terms should be understood according to their ordinary and general meanings.

[0052] The terms "first", "second", "third", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar or like objects or entities, and do not necessarily mean a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchanged under appropriate circumstances.

[0053] The terms "include" and "have" and any variations thereof are intended to cover but not exclusively include, for example, a product or device that includes a series of components, and is not necessarily limited to all components clearly listed, but can include other components that are not clearly listed or inherent to such products or devices.

[0054] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that can perform the functions associated with that element.

[0055] In the embodiments of the present application, the display device 200 generally refers to a device having the ability of picture display and data processing. For example, the display device 200 includes but is not limited to a smart television, a mobile terminal, a computer, a monitor, an advertising screen, a wearable device, a virtual reality device, an augmented reality device, etc.

[0056] Figure 1 A diagram of an operation scenario between a display device and a control device provided for some embodiments of the present application. As shown in Figure 1 A user can operate the display device 200 through a touch operation, a mobile terminal 300, and a control device 100. For example, the control device 100 can be a remote controller, a stylus, a handle, etc.

[0057] The mobile terminal 300 can serve as a control device for performing human-computer interaction between a user and the display device 200. The mobile terminal 300 can also serve as a communication device for establishing a communication connection with the display device 200 and performing data interaction. In some embodiments, the mobile terminal 300 can install a software application with the display device 200, implement connection communication through a network communication protocol, and achieve the purpose of one-to-one control operation and data communication. The mobile terminal 300 can also display audio and video content on the display device 200 to achieve a synchronous display function.

[0058] As shown in Figure 1 It is also shown in the

[0059] The display device 200 can provide a broadcast receiving television function, and can additionally provide an intelligent network television function with computer support, including but not limited to, a network television, a smart television, an Internet protocol television (IPTV), and the like.

[0060] Figure 2 The hardware configuration diagram of the display device 200 is provided in some embodiments of the present application. Figure 1 The hardware configuration diagram of the display device 200 is provided in some embodiments of the present application.

[0061] In some embodiments, the display device 200 can include at least one of a tuning demodulator 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface.

[0062] In some embodiments, the detector 230 is used to collect signals of an external environment or interaction with the outside. For example, the detector 230 includes a light receiver for collecting ambient light intensity, or the detector 230 includes an image collector such as a camera, which can be used to collect an external environment scene, a user attribute, or a user interaction gesture, or the detector 230 includes a sound collector such as a microphone, which is used to receive external sound.

[0063] In some embodiments, the display 260 includes a display functional component for presenting a picture, and a driving component for driving image display. The display 260 is used to receive an image signal output from the controller 250 for display. For example, the display 260 can be used to display video content, image content, and components of a menu control interface, as well as a user control UI interface, and the like.

[0064] In some embodiments, the communication device 220 is a component for communicating with the external device or the server 400 according to various communication protocol types. The display device 200 can be provided with multiple communication devices 220 according to different supported communication manners. For example, when the display device 200 supports wireless network communication, the display device 200 can be provided with a communication device 220 containing WiFi function. When the display device 200 supports Bluetooth connection communication, the display device 200 needs to be provided with a communication device 220 containing Bluetooth function.

[0065] The communication device 220 can connect the display device 200 with the external device or the server 400 through wireless or wired connection. The wired connection can connect the display device 200 with the external device through data line, interface, etc. The wireless connection can connect the display device 200 with the external device through wireless signal or wireless network. The display device 200 can directly establish connection relationship with the external device, or indirectly establish connection relationship through gateway, route, connection device, etc.

[0066] In some embodiments, the controller 250 can include at least one of central processor, video processor, audio processor, graphic processor, power processor, first interface to nth interface for input / output, and the controller 250 controls the operation of the display device and responds to the user's operation through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200.

[0067] In some embodiments, the controller 250 and the tuner demodulator 210 can be located in different split devices, i.e. the tuner demodulator 210 can also be in the external device of the main device where the controller 250 is located, such as external set-top box, etc.

[0068] In some embodiments, the user can input user command through the graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input command through the graphical user interface (GUI).

[0069] In some embodiments, the audio output device 270 can be the native loudspeaker of the display device 200, or can be the audio output device connected to the display device 200. For the audio output device connected to the display device 200, the display device 200 can also be provided with an external audio output terminal, and the audio output device can be connected to the display device 200 through the external audio output terminal to output the sound of the display device 200.

[0070] In some embodiments, the user input interface 280 can be used to receive the instruction input from the user.

[0071] Figure 3 The hardware configuration block diagram of the control device provided in some embodiments of the present application is shown in FIG. 1. As shown in FIG. 1, the control device 100 can include a controller 110, a communication interface 130, a user input / output interface, a memory, and a power supply. Figure 1 Figure 3 The hardware configuration block diagram of the control device provided in some embodiments of the present application is shown in FIG. 1. As shown in FIG. 1, the control device 100 can include a controller 110, a communication interface 130, a user input / output interface, a memory, and a power supply.

[0072] The control device 100 is configured to control the display device 200, and can receive the user's input operation instructions and convert the operation instructions into instructions that the display device 200 can recognize and respond to, thereby playing a role of an intermediary in the interaction between the user and the display device 200.

[0073] In some embodiments, the control device 100 can be a smart device. For example, the control device 100 can install various applications for controlling the display device 200 according to the user's needs.

[0074] In some embodiments, as shown in FIG. 3, the mobile terminal 300 or other smart electronic device can play a similar function of the control device 100 after installing the application for controlling the display device 200. Figure 1

[0075] The controller 110 includes a processor 112 and a RAM 113 and a ROM 114, a communication interface 130, and a communication bus. The controller 110 is used to control the operation and running of the control device 100, and the communication and cooperation between the internal components, and the data processing function of the external and internal.

[0076] The communication interface 130, under the control of the controller 110, realizes the communication of control signals and data signals between the control device 100 and the display device 200. The communication interface 130 can include at least one of a WiFi chip 131, a Bluetooth module 132, an NFC module 133, and other near field communication modules.

[0077] The user input / output interface 140, wherein the input interface includes at least one of a microphone 141, a touchpad 142, a sensor 143, a key 144, and other input interfaces.

[0078] In some embodiments, the control device 100 includes at least one of the communication interface 130 and the input / output interface 140. The control device 100 is configured with a communication interface 130, such as a WiFi, Bluetooth, NFC, etc. module, which can encode the user input instructions through a WiFi protocol, or a Bluetooth protocol, or an NFC protocol, and send them to the display device 200.

[0079] The memory 190 is used to store various running programs, data and applications for driving and controlling the control device 100 under the control of the controller. The memory 190 can store various control signal instructions input by the user. ​​

[0080] A power supply 180 is configured to provide operating power for the components of the display device 200 under the control of the controller.

[0081] To perform user interactions, in some embodiments, the display device 200 can run an operating system. The operating system is a computer program that manages and controls the hardware and software resources of the display device 200. The operating system can provide a user interface to allow the user to interact with the display device 200 and support running various applications.

[0082] It should be noted that the operating system can be a native operating system based on a specific operating platform, a third-party operating system deeply customized based on a specific operating platform, or an independent operating system specially developed for the display device.

[0083] The operating system can be divided into different modules or levels according to the implemented functions, for example, as shown in FIG. 2, in some embodiments, the system is divided into four layers, from top to bottom, the application layer (referred to as the “application layer”), the application framework layer (referred to as the “framework layer”), the system library layer, and the kernel layer. Figure 4

[0084] In some embodiments, the application layer is configured to provide services and interfaces for applications, so that the display device 200 can run the applications and interact with the user based on the applications. At least one application can run in the application layer, which can be a window (Window) program, a system setting program, or a clock program provided by the operating system, or an application developed by a third-party developer. In specific implementation, the applications in the application layer are not limited to the above examples.

[0085] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications. The application framework layer includes some pre-defined functions. The application framework layer is equivalent to a processing center that decides which application in the application layer to act. The application can access the resources in the system and obtain the services of the system through the API interface during execution.

[0086] As shown in FIG. 2, in some embodiments, the framework layer includes a window manager (Window Manager), a resource manager (Resource Manager), a package manager (Package Manager), and a content provider (Content Provider). Figure 4 ​As shown, in the embodiment of the present application, the application framework layer includes a view system, managers, content providers, etc., wherein the view system can design and implement the interface and interaction of the application, and the view system includes lists, grids, text boxes, buttons, etc. The manager includes at least one of the following modules: an activity manager for interacting with all activities running in the system; a location manager for providing system services or applications with access to the system location service; a package manager for retrieving various information related to the application packages currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0087] In some embodiments, the activity manager is used to manage the lifecycle of each application and common navigation back functions, such as controlling application exit, opening, and back. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, taking screenshots, and controlling changes in display windows, such as shrinking, shaking, or distorting the display window.

[0088] In some embodiments, the system runtime layer can provide support for the framework layer. When the framework layer is used, the operating system will run the instruction library contained in the system runtime layer, such as the C / C++ instruction library, to implement the functions to be implemented by the framework layer.

[0089] In some embodiments, the kernel layer is a functional layer between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. Figure 4 As shown, the kernel layer can be configured with hardware drivers, and the drivers included in the kernel layer can be at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0090] It should be noted that the above examples are only a simple division of the operating system functions, and do not constitute a limitation on the specific operating system form of the display device 200 in the embodiments of the present application. According to the function of the display device, the type of the operating system, and other factors, the number and specific type of the levels contained in the operating system can be in other forms.

[0091] In some embodiments, a target detection method is provided, which can be applied to the display device described above, implemented by the display device alone; or applied to a system composed of the display device and a server, and implemented by the display device and the server in interaction; the embodiments of the present application do not limit this, and the embodiments of the present application are described by taking application to the display device described above as an example. The target detection method provided by the embodiments of the present application is described with reference to Figure 5 as shown, comprising the following steps:

[0092] Step 501, using a target detection model to perform target detection on a to-be-detected image to obtain a preliminary detection result, the preliminary detection result including at least one set of target detection data, the target detection data including a target frame, a target type, and a confidence.

[0093] The target detection model can be a visual detection model with relatively small model parameter size. The embodiments of the present application also refer to it as a small model. For example, the target detection model can be a yolo model, and the target detection model includes a backbone network, a neck network, and a prediction head.

[0094] Optionally, the target detection method provided by the embodiments of the present application can be applied to visual detection in a vertical field. For example, it can be applied to fire detection in factory safety, accident detection in urban traffic, abnormal event detection of adding passengers in rail transit, and personnel abnormal behavior analysis in multiple fields, and other application scenarios. In these scenarios, a monitoring camera can be installed, the camera can collect images in real time, and send them to the display device. Each image received by the display device can be used as a to-be-detected image.

[0095] The to-be-detected image can be input to the target detection model, and the target detection model can perform target detection on the to-be-detected image to obtain a preliminary detection result. The preliminary detection result includes at least one set of target detection data, the target detection data including a target frame, a target type, and a confidence. The target frame is used to represent the position of the target in the to-be-detected image, for example, it can be represented by four vertices; the target type is used to represent the type of the target, such as a person, an animal, an object, etc.; and the confidence is used to represent the possibility that the content in the target frame really belongs to the target.

[0096] In step 502, for the first target detection data with the confidence greater than the first preset value and less than the second preset value, a context region image of a target box contained in the first target detection data is obtained, a target detection prompt word is generated based on a target type contained in the first target detection data, and the validity of the first target detection data is determined based on the context region image and the target detection prompt word by using the multi-modal large model.

[0097] For each set of target detection data, the confidence contained in the target detection data can be determined. If the confidence is less than the first preset value, it means that the content in the target box in the target detection data is very unlikely to belong to the target, and the target detection data is directly excluded. If the confidence is greater than the second preset value, it means that the content in the target box in the target detection data is very likely to belong to the target, and the target detection data is directly used as part of the final detection result without subsequent validity re-trial as a doubtful case. If the confidence is greater than the first preset value and less than the second preset value, or equal to the first preset value, or equal to the second preset value, the target detection data is used as the first target detection data, i.e., as a doubtful case for subsequent validity re-trial.

[0098] The process of validity re-trial will be introduced below.

[0099] After determining the first target detection data in the above manner, the target box contained in the first target detection data is expanded to obtain a context region image, a target detection prompt word is generated based on the target type contained in the first target detection data and a pre-set instruction template, the context region image and the target detection prompt word are input into the multi-modal large model, and the validity of the first target detection data is determined based on the output result of the multi-modal large model.

[0100] In step 503, in the case where the first target detection data is valid, the final detection result corresponding to the to-be-detected image is constructed based on the target detection data with the confidence greater than the second preset value and the first target detection data.

[0101] In the case where the first target detection data is valid, the target detection data with the confidence greater than the second preset value and the first target detection data together constitute the final detection result corresponding to the to-be-detected image.

[0102] In step 504, in the case where the first target detection data is invalid, the target detection data with the confidence greater than the second preset value is used as the final detection result corresponding to the to-be-detected image.

[0103] In the case where the first target detection data is invalid, the target detection data with the confidence greater than the second preset value is the final detection result corresponding to the to-be-detected image.

[0104] In some embodiments, referring toFigure 6 As shown, the to-be-detected scene data includes a to-be-detected image and a context region image obtained by the above method. The complete to-be-detected image can be input to the small model. For the small model detection result, that is, the preliminary detection result, the confidence of each group of target detection data is judged. If it is greater than the second preset value, it means that the content in the target frame in the target detection data is very likely to belong to the target, and there is no need to be a suspect case for subsequent effectiveness review. The target detection data is directly used as a high-confidence positive case and becomes part of the final detection result. If it is greater than the first preset value and less than the second preset value, or equal to the first preset value, or equal to the second preset value, the target detection data is used as the first target detection data, that is, a suspect positive case for subsequent effectiveness review. In the effectiveness review process, the context region image can be used for review. For details, see the above.

[0105] In the above embodiment, the target detection model is used to perform target detection on the to-be-detected image to obtain a preliminary detection result. The preliminary detection result includes at least one group of target detection data. The target detection data includes a target frame, a target type, and a confidence. For the first target detection data with a confidence greater than the first preset value and less than the second preset value, a subsequent effectiveness review is performed as a suspect case. The specific process is as follows: obtaining a context region image of the target frame contained in the first target detection data, generating a target detection prompt word based on the target type contained in the first target detection data, determining the effectiveness of the first target detection data based on the context region image and the target detection prompt word, and constructing a final detection result corresponding to the to-be-detected image based on the target detection data with a confidence greater than the second preset value and the first target detection data in the case of effectiveness of the first target detection data. In the case of invalidity of the first target detection data, the target detection data with a confidence greater than the second preset value is used as the final detection result corresponding to the to-be-detected image. First, the target detection model is used for preliminary target detection. For suspect targets, the multi-modal large model is used for further confirmation to avoid slow response and other problems caused by the large size of model parameters when directly using the multi-modal large model for detection. Moreover, the effectiveness review of the suspect target is based on the context region image, and the review result is more accurate compared to the review based on the entire to-be-detected image.

[0106] In some embodiments, the target detection model includes a backbone network, a neck network, and a prediction head. The target detection model is used to perform target detection on the to-be-detected image to obtain a preliminary detection result, including: inputting the to-be-detected image into the backbone network to perform image feature extraction on the to-be-detected image by the backbone network to obtain a feature map; inputting the feature map into the neck network to perform feature enhancement on the feature map by the neck network to obtain an enhanced feature map; and inputting the enhanced feature map into the prediction head to output the preliminary detection result by the prediction head.

[0107] Optionally, the target detection model is a YOLO model, and the YOLO model comprises a backbone, a neck and a head. The to-be-detected image can be input into the backbone first, the backbone extracts image features of the to-be-detected image to obtain a feature map, the feature map is input into the neck, the neck enhances the features of the feature map to obtain an enhanced feature map, and then the enhanced feature map is input into the head, and the head outputs a preliminary detection result.

[0108] In the above embodiment, the network structure of the target detection model is provided, specifically, the target detection model comprises a backbone, a neck and a head, the output of the backbone is the input of the neck, the output of the neck is the input of the head, the to-be-detected image is input into the backbone, the backbone extracts image features of the to-be-detected image to output a feature map, the backbone enhances the features of the feature map to output an enhanced feature map, the head performs target detection based on the enhanced feature map to output a preliminary detection result. The target detection model is used for preliminary target detection, and for suspicious targets, a multi-modal large model is used for further confirmation, thereby avoiding the problem of slow response caused by the large model parameter size when the multi-modal large model is directly used for detection.

[0109] In some embodiments, obtaining the context region image of the target box contained in the first target detection data comprises: obtaining a prediction box corresponding to the first target detection data without non-maximum suppression; obtaining a maximum circumscribed rectangle box of the target box and the prediction box contained in the first target detection data; increasing the length and width of the maximum circumscribed rectangle box by a preset proportion to obtain an input box, and the region in the input box on the to-be-detected image is the context region image.

[0110] Optionally, as described above, the target detection model comprises a backbone, a neck and a head, the algorithm corresponding to the head comprises a non-maximum suppression algorithm, the non-maximum suppression algorithm is used for performing non-maximum suppression on the target box preliminarily detected by the head, so as to obtain the finally output target box, for convenience of description, the target box preliminarily detected by the head is referred to as a prediction box in the embodiment of the application.

[0111] For the first target detection data with a confidence greater than the first preset value and less than the second preset value, a prediction box without non-maximum suppression corresponding to a target box in the first target detection data can be obtained. Then, a maximum circumscribed rectangle box of the target box and the prediction box in the first target detection data is obtained. Finally, the length and width of the maximum circumscribed rectangle box are each increased by a preset proportion to obtain an input box. Optionally, the length and width of the maximum circumscribed rectangle box can each be doubled to obtain the input box. The region in the input box on the to-be-detected image can be a context region image. For example, refer to FIG. 1. Figure 7

[0112] In the above embodiments, considering that the multi-modal large model has a limited visual resolution and the visual attention of the large model is relatively scattered, when the target region accounts for a small proportion of the whole image or the target key feature has few pixels, the large model is difficult to focus on the target region when the whole image is input to the large model for detection. In order to enable the large model to focus on the suspected target region and fully understand the content of the region, for the suspected target with a confidence greater than the first preset value and less than the second preset value, specific implementation manners of how to obtain a context region image are provided. After the context region image is obtained, further confirmation is performed based on the context region image by using the multi-modal large model, and the rejudgment result is more accurate compared with rejudgment based on the whole to-be-detected image.

[0113] In some embodiments, the target detection prompt word is generated based on a target type contained in the first target detection data, including: adding the target type contained in the first target detection data into a pre-set instruction template to obtain the target detection prompt word.

[0114] The pre-set instruction template can be a template for instructing the multi-modal large model to identify whether the input image contains a target of a certain type. For example, the pre-set instruction template can be: please identify whether the input image contains a target of XX type. The target type contained in the first target detection data can be added into the pre-set instruction template to obtain the target detection prompt word.

[0115] The following is an example:

[0116] Suppose that the target type contained in the first target detection data is a crane type, and the pre-set instruction template is: please identify whether the input image contains a target of XX type. The target detection prompt word obtained after the target type contained in the first target detection data is added into the pre-set instruction template is: please identify whether the input image contains a target of the crane type.

[0117] ​In the above embodiments, for the doubtful target whose confidence is greater than the first preset value and less than the second preset value, the target type contained in the first target detection data can also be added to the pre-set instruction template to obtain a target detection prompt word. Subsequently, based on the target detection prompt word and the above context region image, a further confirmation is made by using the multi-modal large model, and the accuracy is higher.

[0118] In some embodiments, the multi-modal large model comprises a modal encoder, a modal mapper, a large language model, and a modal decoder. The effectiveness of the first target detection data is determined by using the multi-modal large model based on the context region image and the target detection prompt word, which comprises: inputting the context region image into the modal encoder to obtain a visual feature vector; inputting the target detection prompt word into the modal encoder to obtain a text feature vector; inputting the visual feature vector into the modal mapper, and mapping the visual feature vector into a text modal vector by the modal mapper; cutting the text feature vector and the text modal vector into feature slices, inputting the feature slices into the large language model, extracting target information by the large language model, inputting the target information into the decoder, and determining the effectiveness of the first target detection data based on the decoding result output by the decoder.

[0119] Wherein, after obtaining the context region image and the target detection prompt word by the method provided in the foregoing embodiments, the context region image and the target detection prompt word can be respectively input into the modal encoder. The modal encoder extracts features from the context region image to obtain a visual feature vector, and extracts features from the target detection prompt word to obtain a text feature vector.

[0120] Wherein, the modal mapper is used to map different modal feature vectors into the same modal. Optionally, the modal mapper can be used to map different modal feature vectors into the text modal. In this case, only the visual feature vector needs to be input into the modal mapper, and the modal mapper maps the received visual feature vector into a text modal vector.

[0121] Wherein, after obtaining the text feature vector and the text modal vector, the text feature vector and the text modal vector can be cut into feature slices, which are input into the large language model to extract target information. Then, the target information is input into the decoder, and the decoder outputs the corresponding answer.

[0122] For example, the target detection prompt word is: please identify whether there is a crane type target in the input image. After inputting the target detection prompt word and the context region image into the multi-modal large model, the decoder outputs the result of yes or no after processing by each network part. In the case of yes output by the decoder, it is determined that the first target detection data is valid, and in the case of no output by the decoder, it is determined that the first target detection data is invalid.

[0123] In the above embodiments, the structure of the multi-modal large model is provided, specifically including a modal encoder, a modal mapper, a large language model, and a modal decoder. After the target detection prompt word and the context region image are input into the multi-modal large model, the decoder outputs the yes or no result after processing by each part. Finally, the validity of the first target detection data can be determined based on the decoding result output by the decoder. This method first uses the target detection model to perform preliminary target detection, and then uses the multi-modal large model to further confirm the suspicious target, thereby avoiding the slow response caused by the large model parameter size when directly using the multi-modal large model for detection.

[0124] In some embodiments, the target detection method provided by the embodiments of the present application further includes the following steps: in the target detection model training process, for a positive sample, when the confidence output by the target detection model is less than a preset threshold, the positive sample is subjected to data enhancement to expand the sample quantity.

[0125] In the detection process of the small model, the real target that is not recalled is often a difficult sample with less features or lacking of complex features in the training process. In order to improve the recall rate of the small model, the difficult case mining method is used in the training process of the small model to enhance these difficult cases. Specifically, in the training process of the small model, for a positive sample, when the confidence output by the small model is less than a preset threshold, the positive sample can be regarded as a difficult case, and the positive sample can be subjected to different image enhancement processing to expand the sample quantity. In addition, in the training process of the small model, the loss weight of the difficult case can be increased to further improve the recall rate of the small model.

[0126] In the above embodiments, in the target detection model training process, for a positive sample, when the confidence output by the target detection model is less than a preset threshold, the positive sample is subjected to data enhancement to expand the sample quantity. Training the target detection model by using the expanded samples can greatly improve the recall rate of the target detection model.

[0127] In some embodiments, the target detection method provided by the embodiments of the present application further includes the following steps: obtaining a scene picture of a preset scene; performing target extraction and region extraction on the scene picture; inputting the scene picture, the extracted target, and the extracted region into a multi-modal large model for preliminary description to obtain target description information, region description information, and global description information; inputting the scene picture, the target description information, the region description information, the global description information, and a detailed description prompt word into the multi-modal large model for detailed description to obtain detailed description information; constructing multi-type task data, each type of task data including related images, task instructions, and answers; and fine-tuning the multi-modal large model based on the detailed description information and the multi-type task data.

[0128] Wherein, in order to further enhance the understanding ability of the multi-modal large model in the vertical field, the multi-modal large model can be fine-tuned by collecting and making vertical field data, and then used in actual scenarios. The vertical field data here includes detailed description information and multi-type task data.

[0129] Optionally, the method of multi-level feature description can be used to construct the detailed description information. Specifically, the scene picture of the preset scene is obtained, and the preset scene can be the scene in which the understanding ability of the multi-modal large model is to be improved. Then, target extraction and region extraction are performed on the scene picture. Optionally, the mature detection and segmentation models can be used for target extraction and region extraction on the scene picture. The scene picture and the pre-set instruction for indicating the multi-modal large model to describe the input image are input into the multi-modal large model, and the multi-modal large model outputs the description information of the scene image, which is referred to as global description information for convenience of description. The extracted target and the pre-set instruction for indicating the multi-modal large model to describe the input image are input into the multi-modal large model, and the multi-modal large model outputs the description information of the extracted target, which is referred to as target description information for convenience of description. Similarly, the extracted region and the pre-set instruction for indicating the multi-modal large model to describe the input image are input into the multi-modal large model, and the multi-modal large model outputs the description information of the extracted region, which is referred to as region description information for convenience of description. For example, see Figure 8 .

[0130] Optionally, as shown in Figure 8 , the extracted target and the extracted region can be filtered first, for example, the target or region with very low confidence is discarded directly, and the subsequent processing is performed on the remaining target and region, so as to obtain the detailed description information.

[0131] Wherein, after obtaining the target description information, the region description information and the global description information, the scene picture, the target description information, the region description information, the global description information and the detailed description prompt word for indicating the multi-modal large model to describe the input image in detail are input into the multi-modal large model, and the multi-modal large model outputs the detailed description information.

[0132] Optionally, as shown in Figure 8 , the detailed description information output by the multi-modal large model can be manually corrected by artificial means to obtain the final detailed description information for fine-tuning the multi-modal large model.

[0133] In addition, multi-type task data can also be constructed, such as: question answering, image-text question answering, detection segmentation, etc., each type of task data includes related images, task instructions and answers, and the language model can be used to expand and rewrite the task instructions and answers to increase diversity, and artificial correction can be performed. The fine-tuning of the multi-modal large model can be performed through the detailed description information and the multi-type task data.

[0134] In the above embodiments, the joint detection of the large model can fully exert the respective advantages. The small model has the advantages of lightweight and fast speed, and is the first step of the joint detection process. All images to be detected will be preliminarily detected by the small model. The multi-modal large model has a strong scene understanding capability, and thus has greater potential to obtain good results when detecting more complex and diverse scenes and targets. However, the multi-modal large model has the disadvantages of a large number of parameters and slow speed. In the joint detection process of the present solution, the multi-modal large model does not detect all targets in the image to be detected, but only judges the suspected positive cases detected by the small model again. Therefore, for the large model, it is necessary to ensure that the real targets are identified in the suspected positive cases, and the missed detection and false detection are reduced to the minimum level, that is, the accuracy is fully guaranteed. The above fine-tuning solution provides an implementation manner of constructing detailed description information and multi-type task data. Based on the fine-tuning of the multi-modal large model based on the detailed description information and the multi-type task data, the understanding of the large model for the vertical scene information can be enhanced, and thus the vertical scene visual detection capability of the multi-modal large model can be improved.

[0135] In some embodiments, a display device is provided, comprising a display and a processor:

[0136] The processor is configured to: perform target detection on the image to be detected by using a target detection model to obtain a preliminary detection result, the preliminary detection result comprising at least one set of target detection data, the target detection data comprising a target frame, a target type and a confidence value; for first target detection data with a confidence value greater than a first preset value and less than a second preset value, obtaining a context region image of the target frame contained in the first target detection data, and generating a target detection prompt word based on the target type contained in the first target detection data; determining the validity of the first target detection data by using a multi-modal large model based on the context region image and the target detection prompt word; in the case that the first target detection data is valid, constructing a final detection result corresponding to the image to be detected based on the target detection data with a confidence value greater than the second preset value and the first target detection data; in the case that the first target detection data is invalid, taking the target detection data with a confidence value greater than the second preset value as the final detection result corresponding to the image to be detected; and the display is configured to display the final detection result corresponding to the image to be detected. For detailed implementation process, please refer to the foregoing description, which will not be repeated here.

[0137] In some embodiments, a computer readable storage medium is provided, having stored thereon a computer program, which, when executed by a processor, implements the target detection method provided in the foregoing, and the detailed implementation process is referred to the foregoing, which will not be repeated here.

[0138] In some embodiments, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the target detection method provided in the foregoing, and the detailed implementation process is referred to the foregoing, which will not be repeated here.

[0139] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

[0140] In order to facilitate explanation, the above description has been made in combination with specific embodiments. However, the above exemplary discussion is not intended to exhaust or limit the embodiments to the specific forms disclosed above. Various modifications and variations can be derived according to the above teachings. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different modified embodiments suitable for specific use considerations.

Claims

1. A target detection method, characterized in that: The method comprises: Performing target detection on the image to be detected using the target detection model to obtain a preliminary detection result, wherein the preliminary detection result includes at least one set of target detection data, and the target detection data includes a target box, a target type, and a confidence level; For first object detection data having a confidence level greater than a first preset value and less than a second preset value, obtaining a context area image of a target frame included in the first object detection data, generating a target detection prompt word based on the target type included in the first object detection data; and determining the validity of the first object detection data using a multimodal large model based on the context area image and the target detection prompt word. If the first target detection data is valid, constructing a final detection result corresponding to the image to be detected based on the target detection data having a confidence level greater than a second preset value and the first target detection data; In the case that the first target detection data is invalid, the target detection data with a confidence level greater than a second preset value is used as the final detection result corresponding to the image to be detected.

2. The method according to claim 1, characterized in that The target detection model includes: a backbone network, a neck network and a prediction head; the target detection model is used to perform target detection on the image to be detected to obtain a preliminary detection result, including: Inputting the image to be detected into the backbone network, and extracting image features from the image to be detected by the backbone network to obtain a feature map; Inputting the feature map into the neck network, and having the neck network perform feature enhancement on the feature map to obtain an enhanced feature map; The enhanced feature map is input into the prediction head, which outputs a preliminary detection result.

3. The method according to claim 1, characterized in that The acquiring of the context area image of the target frame included in the first target detection data includes: Obtaining a prediction box corresponding to the first target detection data without non-maximum suppression; Obtaining a maximum circumscribed rectangular frame of a target frame and the prediction frame included in the first target detection data; The length and width of the maximum circumscribed rectangular frame are increased by a preset ratio to obtain an input frame, and the area within the input frame on the image to be detected is a context area image.

4. The method according to claim 1, wherein Generating a target detection prompt word based on the target type included in the first target detection data includes: The target type included in the first target detection data is added to a preset instruction template to obtain a target detection prompt word.

5. The method according to claim 1, wherein The multimodal large model includes: a modal encoder, a modal mapper, a large language model, and a modal decoder. The determining the validity of the first target detection data based on the context area image and the target detection prompt word using the multimodal large model includes: Inputting the context area image into the modal encoder to obtain a visual feature vector; inputting the target detection prompt word into the modal encoder to obtain a text feature vector; Inputting the visual feature vector into the modality mapper, and having the modality mapper map the visual feature vector into a text modality vector; The text feature vector and the text modality vector are divided into feature slices, the feature slices are input into the large language model, the target information is extracted by the large language model, the target information is input into the decoder, and the validity of the first target detection data is determined based on the decoding result output by the decoder.

6. The method according to claim 1, characterized in that The method further comprises: During the target detection model training process, for positive samples, when the confidence level output by the target detection model is less than a preset threshold, data enhancement is performed on the positive samples to expand the number of samples.

7. The method according to claim 1, characterized in that The method further comprises: Get the scene image of the preset scene; Performing target extraction and region extraction on the scene image; Inputting the scene image, the extracted target, and the extracted region into the multimodal large model for preliminary description, respectively, to obtain target description information, region description information, and global description information; Inputting the scene image, the target description information, the region description information, the global description information, and the detailed description prompt words into the multimodal large model for detailed description to obtain detailed description information; Construct multi-type task data, each type of task data includes relevant images, task instructions and answers; Based on the detailed description information and the multi-type task data, the multimodal large model is fine-tuned.

8. A display device, characterized in that: Including display and processor: The processor is configured to: perform target detection on the image to be detected using the target detection model to obtain a preliminary detection result, wherein the preliminary detection result includes at least one set of target detection data, and the target detection data includes a target box, a target type, and a confidence level; For first target detection data having a confidence level greater than a first preset value and less than a second preset value, obtaining a context area image of a target frame included in the first target detection data, and generating a target detection prompt word based on a target type included in the first target detection data; Determining the validity of the first object detection data using a multimodal large model based on the context area image and the object detection prompt word; If the first target detection data is valid, constructing a final detection result corresponding to the image to be detected based on the target detection data having a confidence level greater than a second preset value and the first target detection data; In the case where the first target detection data is invalid, the target detection data having a confidence level greater than a second preset value is used as the final detection result corresponding to the image to be detected; The display is configured to display a final detection result corresponding to the image to be detected.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images