A target detection method, a detection model training method, an apparatus, and an electronic device

Abstract

The application discloses a target detection method, a detection model training method, a device and electronic equipment, and the method comprises the following steps: extracting initial local region features from a to-be-detected image and extracting global context information from the to-be-detected image; acquiring single-scale context information under multiple scales of the to-be-detected image according to the global context information; determining single-scale context region features of the multiple scales according to each single-scale context information and the initial local region features; connecting the single-scale context region features to obtain multi-scale context region features, and connecting the multi-scale context region features and the initial local region features to obtain target region features; and identifying the target region features to obtain target category information of each target region in the to-be-detected image. The method can improve the accuracy of target detection and further improve the accuracy of small target detection, and can be applied to the technical field of computer vision and the derivative technical field of cloud computing, Internet of Vehicles and the like.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a target detection method, a detection model training method, a device, and an electronic device. Background Technology

[0002] Object detection primarily involves locating objects within images or videos and classifying them. In recent years, significant progress has been made in object detection based on deep convolutional neural network technology, leading to its widespread application in fields such as autonomous driving, complex scene recognition, intelligent search, and intelligent authentication. For example, intelligent vehicles need to detect obstacles before making control decisions; intelligent interaction systems need to detect the person requiring interaction before recognizing gestures and commands; and game testing requires detecting various virtual objects appearing in complex interface scenes to facilitate subsequent detection of their action commands or real-time status information.

[0003] However, current methods typically rely on local visual features in the target image or video for detection, such as extracting ROI (Regions of Interest) regions. This approach usually only achieves good detection results when there is a large difference between local features and global background features. When the difference between the local features and the global background is small, such as in a dark scene, the target object in each frame of the target video has similar color and brightness to the background of the entire image and the boundary is not obvious, it is difficult to detect the target based on the ROI features of local regions, resulting in low detection accuracy. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a target detection method, a detection model training method, an apparatus, and an electronic device with high detection accuracy, so as to improve the detection accuracy of target objects in some complex scenarios.

[0005] One aspect of this invention provides a target detection method, which includes the following steps:

[0006] Acquire the image to be detected;

[0007] Initial local region features are extracted from the image to be detected, and global context information is extracted from the image to be detected.

[0008] The multi-scale context information of the image to be detected is obtained based on the global context information; wherein, the multi-scale context information includes single-scale context information at multiple different scales;

[0009] Based on each single-scale context information and the initial local region features, determine single-scale context region features at multiple different scales;

[0010] The single-scale context region features are concatenated to obtain multi-scale context region features, and the multi-scale context region features are concatenated with the initial local region features to obtain target region features.

[0011] The target region features are identified to obtain the target category information corresponding to each target region in the image to be detected.

[0012] In one possible implementation, extracting global context information from the image to be detected includes:

[0013] Based on a preset downsampling rate, visual features are extracted from the image to be detected using a deep residual network;

[0014] Based on the channel dimension, spatial height, and spatial width determined by the visual feature extraction, the convolutional feature map output by the deep residual network is obtained.

[0015] The global context information is determined based on the convolutional feature map.

[0016] In one possible implementation, obtaining the multi-scale context information of the image to be detected based on the global context information includes:

[0017] Configure the scale size corresponding to each single-scale context information;

[0018] Based on each scale size, the regions in the global context information corresponding to the current scale size are pooled and aggregated to obtain the single-scale context information corresponding to the current scale size;

[0019] The multi-scale context information is determined based on the single-scale context information corresponding to all scale sizes.

[0020] In one possible implementation, the step of pooling and aggregating the regions corresponding to the current scale size in the global context information according to each scale size to obtain the single-scale context information corresponding to the current scale size includes:

[0021] Sequentially obtain the size of each scale as the current scale size;

[0022] Based on the current scale, determine the region to be pooled from the global context information;

[0023] By performing max pooling or average pooling, the region to be pooled is aggregated to obtain the single-scale context information corresponding to the current scale size.

[0024] In this context, the information at each location in the single-scale context information represents the information of all locations in the region to be pooled.

[0025] In one possible implementation, determining multiple single-scale context region features at different scales based on each of the single-scale context information and the initial local region features includes:

[0026] Determine the convolutional feature map corresponding to each of the single-scale context information;

[0027] Calculate the influence value of each position in the convolutional feature map on the initial local region features;

[0028] Based on the influence value and the representation vectors at each position in the convolutional feature map, context aggregation calculation is performed to obtain the single-scale context region features corresponding to the convolutional feature map.

[0029] In one possible implementation, calculating the influence value of each position in the convolutional feature map on the initial local region features includes:

[0030] The representation vectors at each position in the convolutional feature map are subjected to a first dimensionality reduction process, and the representation vectors of the initial local region features are subjected to a second dimensionality reduction process.

[0031] Based on the spatial height and spatial width of the convolutional feature map, a normalization factor is constructed;

[0032] The results of the first dimensionality reduction process and the second dimensionality reduction process are normalized according to the normalization factor to determine the influence value of each position in the convolutional feature map on the initial local region features.

[0033] In one possible implementation, the step of performing context aggregation calculation based on the influence value and the representation vectors at each position in the convolutional feature map to obtain the single-scale context region features corresponding to the convolutional feature map includes:

[0034] When the initial local region extracted from the image to be detected is one, the influence value is multiplied by the representation vector of each position in the convolutional feature map to obtain the region feature vector of each position; the region feature vector of each position is combined to obtain the single-scale context region feature corresponding to the convolutional feature map.

[0035] When there are multiple initial local regions extracted from the image to be detected, the influence value is multiplied by the representation vector of each local region in the convolutional feature map to obtain the region feature vector of each local region; the region feature vectors of each local region are combined to obtain the single-scale context region feature corresponding to the convolutional feature map.

[0036] On the other hand, embodiments of the present invention also disclose a target detection method, which includes the following steps:

[0037] In response to a detection command, an image to be detected is acquired and sent to a target server, so that the target server performs target detection on the image to be detected and identifies the target category information corresponding to each target region in the image to be detected.

[0038] Receive the target category information identified by the target server and display the target detection results;

[0039] The target category information is determined according to the target detection method provided in the first aspect embodiment.

[0040] On the other hand, embodiments of the present invention also disclose a detection model training method, which includes the following steps:

[0041] Obtain the image training set;

[0042] Initial local region features are extracted from each sample image in the image training set, and global context information is extracted from each sample image.

[0043] The multi-scale context information of the sample image is obtained based on the global context information; wherein, the multi-scale context information includes single-scale context information at multiple different scales;

[0044] Based on each single-scale context information and the initial local region features, determine single-scale context region features at multiple different scales;

[0045] The single-scale context region features are concatenated to obtain multi-scale context region features, and the multi-scale context region features are concatenated with the initial local region features to obtain target region features.

[0046] The features of the target region are identified to obtain the predicted category information corresponding to each target region in the sample image;

[0047] Based on the predicted category information and the correct category information of each target region in the sample image, the loss value of the predicted category information is calculated, and the parameters of the detection model are corrected based on the loss value.

[0048] On the other hand, embodiments of the present invention also disclose a target detection device, which includes:

[0049] The first module is used to acquire the image to be detected;

[0050] The second module is used to extract initial local region features from the image to be detected and to extract global context information from the image to be detected.

[0051] The third module is used to obtain multi-scale context information of the image to be detected based on the global context information; wherein, the multi-scale context information includes single-scale context information at multiple different scales;

[0052] The fourth module is used to determine single-scale context region features at multiple different scales based on each of the single-scale context information and the initial local region features;

[0053] The fifth module is used to connect the single-scale context region features to obtain multi-scale context region features, and connect the multi-scale context region features with the initial local region features to obtain target region features;

[0054] The sixth module is used to identify the features of the target region and obtain the target category information corresponding to each target region in the image to be detected.

[0055] On the other hand, embodiments of the present invention also disclose a target detection device, which includes:

[0056] The seventh module is used to respond to a detection command, acquire an image to be detected, and send the image to be detected to the target server, so that the target server performs target detection on the image to be detected and identifies the target category information corresponding to each target region in the image to be detected;

[0057] The eighth module is used to receive the target category information identified by the target server and display the target detection results;

[0058] The target category information is determined according to the target detection method described above.

[0059] On the other hand, embodiments of the present invention also disclose a detection model training device, the device comprising:

[0060] The ninth module is used to acquire the image training set;

[0061] The tenth module is used to extract initial local region features from each sample image of the image training set, and to extract global context information from each sample image;

[0062] The eleventh module is used to obtain multi-scale context information of the sample image based on the global context information; wherein, the multi-scale context information includes single-scale context information at multiple different scales;

[0063] The twelfth module is used to determine multiple single-scale context region features at different scales based on each of the single-scale context information and the initial local region features;

[0064] The thirteenth module is used to connect the single-scale context region features to obtain multi-scale context region features, and connect the multi-scale context region features with the initial local region features to obtain target region features.

[0065] The fourteenth module is used to identify the features of the target region and obtain the predicted category information corresponding to each target region in the sample image;

[0066] The fifteenth module is used to calculate the loss value of the predicted category information based on the predicted category information and the correct category information of each target region in the sample image, and to correct the parameters of the detection model based on the loss value.

[0067] On the other hand, embodiments of the present invention also disclose an electronic device, including a processor and a memory; the memory is used to store a program; the processor executes the program to implement the target detection method as described above or the detection model training method as described above.

[0068] Furthermore, embodiments of the present invention also disclose a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned target detection method or detection model training method.

[0069] This invention extracts initial local region features from the image to be detected and extracts global context information from the image to be detected. When performing target detection, this invention combines the global context information of the image. Building upon existing target recognition methods that only use local region features, it can refer to other global information beyond the local region for target detection, not being limited to local feature information, thus improving detection accuracy. Furthermore, this invention obtains multi-scale context information of the image to be detected based on the global context information. Based on each single-scale context information and the initial local region features, multiple single-scale context region features at different scales are determined. These single-scale context region features are concatenated to obtain multi-scale context region features. These multi-scale context region features are then concatenated with the initial local region features to obtain target region features. The target region features are then identified to obtain target category information corresponding to each target region in the image to be detected. Compared to the target detection process that combines global context information with local region features, this invention further obtains single-scale context region features at different scales based on the global context information, enabling the utilization of features at different spatial scales for target detection. When the target to be detected is a small target, the multi-scale context region features of this invention can more accurately detect small targets, further improving target detection accuracy. Attached Figure Description

[0070] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0071] Figure 1 This is a schematic diagram of the implementation environment for target detection in an embodiment of the present invention;

[0072] Figure 2 This is a schematic flowchart of the target detection method provided in an embodiment of the present invention;

[0073] Figure 3 This is a schematic diagram illustrating the process of extracting global context information provided in an embodiment of the present invention;

[0074] Figure 4 This is a schematic diagram illustrating the process of obtaining multi-scale context information provided in an embodiment of the present invention;

[0075] Figure 5 This is a schematic diagram illustrating the process of obtaining single-scale context information provided in an embodiment of the present invention;

[0076] Figure 6This is a schematic diagram illustrating the process of determining single-scale context region features at multiple different scales provided in an embodiment of the present invention.

[0077] Figure 7 This is a schematic diagram illustrating the process of calculating the influence value provided in the embodiments of the present invention;

[0078] Figure 8 This is a schematic diagram illustrating the process of obtaining single-scale contextual region features based on influence values ​​in an embodiment of the present invention.

[0079] Figure 9 This is a schematic diagram of the complete architecture of target detection in an embodiment of the present invention;

[0080] Figure 10 This is a schematic diagram of the game interface in an embodiment of the present invention;

[0081] Figure 11 This is a schematic diagram of the steps of a detection model training method in an embodiment of the present invention;

[0082] Figure 12 This is a schematic diagram of the implementation environment for a game testing scenario in an embodiment of the present invention. Detailed Implementation

[0083] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0084] It is understood that the terms “first,” “second,” etc., used in this application may be used to describe various concepts herein, but unless otherwise stated, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another.

[0085] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0086] Before providing a detailed description of the embodiments of the present invention, necessary explanations will be given for the technical terms that may be involved in the embodiments of the present invention:

[0087] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0088] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning. The display device with image acquisition components shown in this application mainly relates to computer vision, machine learning / deep learning, autonomous driving, and intelligent transportation.

[0089] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learn-by-doing.

[0090] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image semantic segmentation, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), autonomous driving, intelligent transportation, and other technologies, as well as common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0091] The role of the self-attention mechanism is to better learn the dependencies between global features. The self-attention mechanism obtains the global geometric features of the graph structure in one step by directly calculating the relationship between any two nodes in the graph structure. The self-attention mechanism uses the attention mechanism to calculate in three stages: (1) Introducing different functions and calculation mechanisms to calculate the similarity or relevance between the query and a certain keyword or key. The most common methods include: calculating the vector dot product of the two, calculating the vector similarity between the two, or calculating by introducing an additional neural network; (2) Introducing the calculation method of the activation function (Softmax) to convert the score of the first stage into a numerical value. On the one hand, it can be normalized to organize the original calculated score into a probability distribution in which the sum of the weights of all elements is 1; on the other hand, the weight of important elements can be highlighted more through the internal mechanism of Softmax; (3) The calculation result of the second stage is the corresponding weight coefficient, and then the attention value can be obtained by weighted summation.

[0092] Based on this, the feature concepts that may be involved in the invention will be explained:

[0093] Scale-invariant feature transform (SIFT) is a technique used to describe the different appearances of objects in nature depending on the observation scale. The blurriness of images increases progressively at different scales, mimicking the formation of an object on the retina as the distance increases. Therefore, the larger the scale, the blurrier the image. Image scale space is a local feature that can detect key points in an image. SIFT feature extraction consists of two parts: finding key points in the image and extracting information from the neighborhood of those key points. During feature extraction, only stable key points and their surrounding information are considered, making the features more descriptive.

[0094] Multi-scale information refers to the different information presented after sampling a signal at different granularities. Typically, we can observe different features at different scales, thereby accomplishing different tasks. Generally speaking, smaller / denser granular sampling information reveals more details, while larger / sparser granular sampling information reveals the overall trend.

[0095] Semantic / spatial context information is used to identify and process new targets by capturing the interaction information between different objects and the interaction information between objects and the scene.

[0096] In machine vision and image processing, the region of interest (ROI) is the area in the image that needs to be processed, outlined using shapes such as rectangles, circles, ellipses, or irregular polygons.

[0097] Based on the aforementioned theoretical foundations, and with the research and advancements in artificial intelligence technology, AI technology has been studied and applied in multiple fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI technology will be applied in more fields and play an increasingly important role.

[0098] It is understood that the target detection method provided in this embodiment of the invention can be applied to any computer device with data processing and computing capabilities, and this computer device can be various types of terminals or servers. When the computer device in the embodiment is a server, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. Optionally, the terminal can be a smartphone, tablet computer, laptop computer, or desktop computer, but it is not limited to these.

[0099] It should be further noted that the terminals involved in the embodiments of the present invention include, but are not limited to, smartphones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, and aircraft. The embodiments of the present invention can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0100] In some possible implementations, a computer program that can implement the target detection method or a target detection model training method provided in the embodiments of this application can be deployed and executed on a computer device, or executed on multiple computer devices located in one location, or executed on multiple computer devices distributed in multiple locations and interconnected through a communication network. Multiple computer devices distributed in multiple locations and interconnected through a communication network can form a blockchain system.

[0101] Taking the target detection process in a game scene as an example, based on the fact that multiple computer devices can form a blockchain system, the computer device implementing the target detection method in this embodiment of the invention can be a node in the blockchain system. This node stores a machine learning model. This machine learning model acquires single-scale contextual information of the game interface screenshot of the target object at different scales. When acquiring single-scale contextual region features at different scales, it provides information not included in the original local region features through the single-scale contextual information at different scales, and combines this with the original local region features to determine the target region features, thereby identifying the target category information corresponding to each target region on the game interface screenshot. This node, or nodes corresponding to other devices in the blockchain, can also store game interface screenshots, target type information, and other intermediate feature data obtained during the prediction process, etc.

[0102] like Figure 1 The diagram shown is a schematic representation of an implementation environment provided by an embodiment of the invention. (Refer to...) Figure 1The implementation environment includes at least one terminal 101 and a server 102. Terminal 101 and server 102 can connect wirelessly or via a wired network to exchange data. Terminal 101 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. Terminal 101 has an application that supports image display installed and running. Server 102 is the backend server for the object detection application, or a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms.

[0103] exist Figure 1Based on the implementation environment shown, this embodiment of the invention also provides a training scenario for an object detection model. This object detection model improves the accuracy of detection results through a Double Module Global Context Network (DMGCN). This DMGCN includes a Space Context Representation Module (SCRM) and at least one Space Context Dependent Module (SCDM). The Space Context Representation Module is mainly used to construct multi-scale context representations. This module can obtain the global context (i.e., multi-scale context) from the previous processing step, and perform average pooling on the multi-scale context according to the principle of pyramid scaling to obtain a single-scale context representation. It should be noted that the scaling transformation during the multi-scale context representation process can be the pyramid scaling transformation or other feasible scaling methods, which are not enumerated here. The spatial context dependency module primarily calculates dependency based on the single-scale context representation output by the spatial context representation module and the pre-processed Region of Interest (ROI). Simultaneously, it aggregates context based on the ROI to generate single-scale context ROI features, and then fuses these single-scale context ROI features to obtain multi-scale context ROI features. Further, based on the dependency calculation results, the parallel multi-scale context ROI features and the pre-processed ROI features are concatenated for classification. It should be noted that in this embodiment, dependency calculation can be affinity calculation or other weight calculation processes that reflect the degree of influence between feature information. The global context obtained by the spatial context representation module can be a convolutional feature map obtained by using, for example, a Convolutional Neural Network (CNN) model to extract visual features from the image. This convolutional feature map can represent contextual information. After obtaining the convolutional feature map, the embodiment can identify candidate features in the convolutional feature map using a Region Proposal Network (RPN) and select targets by bounding boxes to obtain the Region of Interest (ROI), which serves as one of the inputs to the spatial context dependency module.

[0104] It should be further explained that in this scenario, the server stores a large number of historical game interface screenshots uploaded by game terminals during gameplay. Multiple historical game interface screenshots are combined to form an image training set, which is then input into the object detection model. The object detection model first extracts global contextual information from each sample image in the image training set to represent the relationship between each pixel and its surrounding pixels, and extracts features of the region of interest (ROI) in each sample image. Then, the global contextual information is input into the spatial context representation module, which outputs single-scale contextual information representing multiple different scales. All single-scale contextual information and ROI features are then input into the spatial context dependency module, which outputs single-scale contextual region features representing multiple different scales. The ROI features are then concatenated based on all single-scale contextual region features to obtain the target region features. The category corresponding to the target region features is then predicted, and a loss value is calculated based on the predicted category. The parameters of the object detection model are then optimized and adjusted based on the loss value, resulting in the trained object detection model.

[0105] exist Figure 1 Based on the implementation environment shown, this embodiment of the invention also provides a scenario for object detection. In this scenario, after obtaining object authorization from the game test player, the server interacts with the terminal device running the game client to obtain a screenshot of the game interface when the game is running. A pre-trained object detection model is stored on the server. The game interface screenshot is input into the object detection model to extract global contextual information and multi-scale contextual region features from the screenshot. This provides multi-scale feature information during object classification, thereby improving the accuracy of the classification results.

[0106] like Figure 2 The diagram shown is a flowchart of a target detection method provided in an embodiment of the present invention. The execution subject of this target detection method can be any of the aforementioned computer devices. (Refer to...) Figure 2 The method includes the following steps:

[0107] S201. Obtain the image to be detected.

[0108] The image to be detected can be a game interface image, a vehicle image, a face image, or an image of an animal or plant, or other types of images. For example, taking a game testing scenario, the target object can connect different models of game terminals to the detection terminal via a connection cable. The target object installs the game application to be tested on the different models of game terminals through the detection terminal. After the game application is installed, the detection terminal controls the game application installed on the game terminal to launch. In response to the game detection command, the detection terminal controls the game application to jump to the interface to be detected. The detection terminal controls the game terminal to take a screenshot of the interface to be detected, obtaining the image to be detected. It should be noted that the game interface image captured by the game terminal can be a single frame or multiple frames. After capturing the image frames, the game terminal can transmit them wirelessly to the detection terminal. When performing target detection, the detection terminal uses the acquired frame image as the image to be detected.

[0109] S202. Extract initial local region features from the image to be detected, and extract global context information from the image to be detected.

[0110] In this context, a local region can be understood as a Region of Interest (RoI) or a proposed region of interest in the original image. Local region features refer to the image features within that RoI. For example, when a game interface screenshot includes multiple game objects such as heroes and minions, one or more of the target game objects can be marked with bounding boxes as RoIs. The local region features refer to the image features within those target game objects. In this embodiment, the image to be detected can be input into a convolutional neural network, which outputs the image features of the image to be detected. Then, a Region Proposal Network (RPN) is used to determine the local regions on the image to be detected as RoIs, and the image features corresponding to these RoIs are used as the local region features.

[0111] In the field of computer vision, contextual information can include semantic contextual information, spatial contextual information, and scale contextual information. Semantic and spatial contextual information can both be understood as interaction information between different objects and interaction information between objects and the scene. For example, in a game scene, in a single frame of a game interface screenshot, global contextual information includes interaction information between different game objects in that screenshot and interaction information between game objects and the background.

[0112] For example, in a game scenario, when a game terminal is running a game, it takes a screenshot of the target interface and sends it to a server. The server pre-stores a Convolutional Neural Network (CNN). This CNN performs convolution operations on the target game interface screenshot to extract features from it. Specifically, the convolutional layers within the CNN continuously perform sliding operations on the target game interface screenshot through convolution processing to obtain the corresponding inner product results. The maximum value of each local block in the inner product result is then used for pooling, thereby obtaining features from different local regions corresponding to the target game interface screenshot.

[0113] S203. Obtain multi-scale context information of the image to be detected based on global context information.

[0114] In this context, image scale refers to the coarseness or fineness of the image content. Scale is used to simulate the distance of a target object from another object. Understandably, the farther away the target object is, the more likely it is to appear as only a rough outline; the closer the target object is, the more likely it is to reveal detailed information. For example, in a road condition detection scenario, in a vehicle obstacle detection scenario, an onboard camera acquires images of the area in front of the vehicle in real time. After converting this image into a frequency domain image, the coarseness or fineness of the image represents the low-frequency and high-frequency components of the image's frequency domain information. A coarse image represents information concentrated mostly in the low-frequency band, with a small amount of high-frequency information. A fine image represents rich information components, containing information in both high and low frequency bands. The scale space of an image refers to the set of different scales of the same image. Multiple scales refer to different spatial sizes. This embodiment extracts global contextual information from the image to be detected at different scales to obtain contextual information at different scales. For example, taking a vehicle image as an example, different scales include 1.0, 2.0 and 3.0 respectively. Then, the single-scale context information corresponding to the vehicle image at scale 1.0 is extracted, the single-scale context information corresponding to the vehicle image at scale 2.0 is extracted, and the single-scale context information corresponding to the vehicle image at scale 3.0 is extracted. The single-scale context information corresponding to scales 1.0, 2.0 and 3.0 are combined into multi-scale context information, so that context information can be extracted from different scales.

[0115] S204. Based on each single-scale context information and the initial local region features, determine multiple single-scale context region features at different scales, connect the single-scale context region features to obtain multi-scale context region features, and connect the multi-scale context region features with the initial local region features to obtain target region features.

[0116] The process of concatenating single-scale context features can be achieved using the `concat()` function to join two or more pieces of information. The `concat()` function does not alter the existing information; it merely generates a concatenated copy. In this embodiment, after obtaining multiple single-scale context features and initial local region features, multiple context features at different scales can be generated based on the relationship between these two pieces of information. Then, `concat()` is used to connect the various single-scale context region features into multi-scale context region features. This allows the multi-scale context region features to provide features beyond the initial local region features, resulting in more accurate target region features. In this embodiment, the multi-scale context region features and the original local region features have the same channel dimension.

[0117] S205. Identify the features of the target region to obtain the target category information corresponding to each target region in the image to be detected.

[0118] In step S204, after obtaining the target region features, these features are input into a fully connected layer to classify the target region. It is understood that all neurons in the fully connected layer are connected by weights. Once the preceding convolutional layers have captured sufficient features, the fully connected layer can classify the target object to be detected. It should be noted that the fully connected layer in this embodiment includes a linear layer with a ReLU activation function for label prediction.

[0119] To improve the accuracy of global context information extraction, in some feasible embodiments, such as Figure 3 As shown, the step of extracting global context information from the image to be detected may include steps S2021-S2023:

[0120] S2021. Based on the preset downsampling rate, visual features are extracted from the image to be detected using a deep residual network.

[0121] Downsampling refers to reducing or downsampling an image. The downsampling rate refers to the factor by which the image is reduced. For example, taking a target person image captured by a camera as an example, if the target person image has a size of M*N, then when the preset downsampling rate is 2, the downsampled target image will have a size of (M / 2)*(N / 2). In this embodiment, to obtain global context information, the downsampling rate can be set as the stride between the image to be detected and the global context information. Then, a Deep Residual Network (ResNet) is used to extract visual features from the image to be detected. It can be understood that a Deep Residual Network has many bypass branches that directly connect the input to subsequent layers, allowing those layers to directly learn the residuals. For example, visual features may include information such as color, contour, texture, and spatial relationships in the image to be detected.

[0122] S2022. Based on the channel dimension, spatial height, and spatial width determined by visual feature extraction, the convolutional feature map output by the deep residual network is obtained.

[0123] In this context, channels are used for image feature detection, and the strength of a channel value reflects the strength of the current feature. Channel dimension refers to the number of input channels of the convolutional kernel in a deep residual network; spatial height refers to the height of the convolutional kernel; and spatial width refers to the width of the convolutional kernel. The number of input channels corresponds to the matrix dimension of the input image. Convolutional feature maps can be used to represent the relationships between pixels in the global context.

[0124] S2023. Determine global context information based on the convolutional feature map.

[0125] For example, in a scenario where a target object is identified while a vehicle is in motion, an in-vehicle terminal acquires images of the surrounding environment of the target vehicle during its journey. These images are then sent to a server, which pre-stores a deep residual network. The deep residual network extracts visual features from the surrounding environment image based on the stride D between the surrounding environment image and the global context. The stride D can also be understood as the downsampling rate of the deep residual network. Then, based on the visual feature extraction process, the deep residual network is set to have a channel dimension of C, a spatial height of H, and a spatial width of W. After inputting the surrounding environment image into the deep residual network, a convolutional feature image X∈R representing the global context is output. C×H×W In this case, each position in the convolutional feature image X may represent a pixel containing a D×D surrounding environment image. Then, based on the pixel that each position may represent, global context information in the surrounding environment image is determined, thereby providing other local region feature information in addition to local region features when performing target recognition.

[0126] To extract more contextual information corresponding to different scales, in some feasible embodiments, such as Figure 4 As shown, the step of obtaining multi-scale context information of the image to be detected based on global context information may include steps S2031-S2033:

[0127] S2031. Configure the scale size corresponding to each single-scale context information;

[0128] In this structure, the scale corresponding to each single-scale contextual information can be represented by the scale of each layer in the pyramid. The pyramid structure is used in the forward convolution process of the network. For each resolution feature map, a feature map scaled by a factor of two from the next resolution is introduced and summed element-wise from bottom to top. This allows the high-resolution, low-semantic-information low-level feature maps and low-resolution, high-semantic-information high-level feature maps in the convolutional neural network to be fused, resulting in a fused feature map containing more semantic information. Image resolution refers to the degree of quantization of an image in the horizontal and vertical directions, which can also be understood as the level of detail an image can display. When the size of objects within an image is small or the contrast is low, a higher resolution is required to observe the image details.

[0129] S2032. Based on each scale size, pool and aggregate the regions in the global context information corresponding to the current scale size to obtain the single-scale context information corresponding to the current scale size.

[0130] Pooling aggregation can include two processes: pooling and aggregation. Pooling involves averaging the values ​​of a region in the (intermediate) feature image at the current scale as the representation of that region. Aggregation, after obtaining the average representation of the region through pooling, aggregates all content information within that region to a target location within that region, thus obtaining single-scale context information at the current scale. For example, in a vehicle detection scenario, where the scale is determined to be 2, the global context information obtained in the previous steps is first acquired. This global context information may include all vehicle body information and license plate information of the target vehicle. Based on the predetermined scale information, the feature map containing this global context information is divided into several 2x2 image regions. For example, in an image region containing local vehicle body features, the region is first averaged to calculate the average pixel value, which is then used as the pixel representation value of that region. Simultaneously, to preserve as much content information as possible within the region, the vehicle body feature information within that region is aggregated to a specific location within that region, thereby obtaining single-scale context information for the target vehicle at a scale of 2.

[0131] S2033. Determine multi-scale context information based on the single-scale context information corresponding to all scale sizes.

[0132] For example, in a vehicle detection scenario, when all scale sizes are 2, 4, and 8, pooling operations are performed on the corresponding 2*2, 4*4, and 8*8 regions in the global context information to obtain multiple single-scale context information corresponding to the 2*2, 4*4, and 8*8 regions. The multiple single-scale context information are then concatenated to obtain multi-scale context information.

[0133] To improve the accuracy of target detection and recognition, in some feasible embodiments, such as Figure 5 As shown, the step of pooling and aggregating the regions corresponding to the current scale size in the global context information according to each scale size to obtain the single-scale context information corresponding to the current scale size may include steps S20321-S20322:

[0134] S20321. Sequentially obtain the size of each scale as the current scale size, and determine the region to be pooled from the global context information based on the current scale size;

[0135] The pooling region can refer to target regions of different scales obtained by dividing the data according to different scales. For example, in an animal and plant detection scenario, since the shapes and sizes of different kinds of animals and plants vary in the feature map of the global context information, it is necessary to determine the pooling region by dividing it into different scales. Each scale can include 3, 6, and 9. If the scale is 3, then a 3*3 region can be determined in the global context information as the pooling region. The pooling region at this scale can cover smaller animals and plants, such as mosses, ferns, and insects. Similarly, if the scale is 6, then a 6*6 region can be determined in the global context information as the pooling region. If the scale is 9, then a 9*9 region can be determined in the global context information as the pooling region. Correspondingly, in a larger pooling region, it can cover, for example, shrubs or animals such as cats and dogs.

[0136] S20322. By performing max pooling or average pooling, the region to be pooled is aggregated to obtain the single-scale context information corresponding to the current scale.

[0137] In this contextual information, the information at each location corresponds to the information at all locations in the region to be pooled. Pooling refers to a downsampling process that maintains invariance to operations such as translation and scaling. Max pooling uses the maximum value of a region as its representation. Max pooling can suppress the phenomenon of mean shift in estimation caused by network parameter errors, and better extract texture information within the image to be detected. Mean pooling uses the average value of a region as its representation, which can suppress the phenomenon of increased variance in estimation caused by limited neighborhood size, and preserves the background better.

[0138] In this embodiment, pooling of adjacent overlapping regions and spatial pyramid pooling can also be applied, ensuring that overlapping regions exist during each window swipe. Pooling of adjacent overlapping regions refers to using a step size smaller than the window width, while spatial pyramid pooling is based on multi-scale information. For example, pooling of 1*1, 2*2, and 4*4 matrices is calculated simultaneously, and the results are concatenated as input to the next network layer.

[0139] For example, in a vehicle detection scenario, features containing global contextual information of the image of the vehicle to be detected are acquired. Figure X Then, the feature map is processed using a pyramid to obtain a multi-scale context representation. In the pyramid scaling, 's' represents the scale of each level, and this value can take different values ​​such as 1, 2, 3, and 6. One of these values ​​is selected as the scale 's', and the feature map is then... Figure X The position of s*s is taken as a region, and the average value in this region is used as the output to obtain the single-scale feature. Figure X s. It is understandable that when the scale is s, average pooling yields a single-scale feature. Figure X The dimension of s is R C×(H / s)×(W / s) The contents of regions s*s are aggregated to a single location to obtain a single-scale context representation X∈R. C×(H / s)×(W / ) In this single-scale feature Figure X In s, each position represents a (D×s)*(D×s) region in the image of the vehicle to be detected. Based on this processing, single-scale context representations corresponding to other scales are obtained. Multiple single-scale context representations are concatenated to obtain multi-scale context representations, thereby enabling the capture of visual feature information with multiple size variations when performing vehicle target detection.

[0140] In some feasible embodiments, such as Figure 6 As shown, the step of determining single-scale contextual region features at multiple different scales based on each single-scale contextual information and the initial local region features may include steps S2041-S2043:

[0141] S2041. Determine the convolutional feature map corresponding to each single-scale context information;

[0142] The process involves inputting the image to be detected into a convolutional neural network (CNN), which outputs a convolutional feature map. When the image to be detected is transformed at each single scale, and the scale-transformed images are then input into the CNN, a convolutional feature map corresponding to the contextual information at each single scale is output.

[0143] S2042. Calculate the influence value of each position in the convolutional feature map on the initial local region features;

[0144] The influence value can be used to characterize the degree of interaction between two types of information when they are combined. For example, in a target object detection scenario, after inputting the image to be detected into a convolutional neural network, a convolutional feature map and initial local region features can be output. It can be understood that the convolutional feature map contains this local region. By calculating the degree of interaction between this local region feature and various positions on the convolutional feature map, the influence value between the local region feature and the convolutional feature map can be obtained.

[0145] S2043. Perform context aggregation calculation based on the influence value and the representation vector at each position in the convolutional feature map to obtain the single-scale context region features corresponding to the convolutional feature map.

[0146] Here, the representation vector is used to represent the context representation of the corresponding position on the convolutional feature map; specifically, in the embodiment, this representation vector can refer to the single-scale feature map after pooling aggregation. Figure X The vector representation of the information content aggregated at each position in s. After obtaining the influence values ​​of local region features at each position on the convolutional feature map, the relationship between the initial local region features and the convolutional feature map can be redistributed using the influence values ​​and representation vectors. Based on the redistributed relationship, the context is aggregated and calculated to obtain the single-scale context region features corresponding to the convolutional feature map.

[0147] In some feasible embodiments, such as Figure 7 As shown, the step of calculating the influence of each position in the convolutional feature map on the initial local region features may include steps S20421-S20423:

[0148] S20421. Perform a first dimensionality reduction on the representation vectors at each position in the convolutional feature map, and a second dimensionality reduction on the representation vectors of the initial local region features.

[0149] Dimensionality reduction refers to projecting high-dimensional data into a low-dimensional space without altering the data structure. Reducing the dimensionality of high-dimensional data before processing effectively lowers computational costs. Essentially, the first dimensionality reduction process involves projecting the representation vectors from high-dimensional to low-dimensional space after obtaining the representation vectors of each position in the convolutional features; similarly, the second dimensionality reduction process involves projecting the representation vectors from high-dimensional to low-dimensional space after obtaining the representation vectors of the initial local region features.

[0150] S20422. Construct a normalization factor based on the spatial height and spatial width of the convolutional feature map;

[0151] Convolutional feature maps are obtained by processing the image to be detected using a convolutional neural network. Spatial height and spatial width can be understood as the height and width of the input or output channels in the convolutional neural network. Normalization refers to transforming values ​​within a certain range to a target range. The purpose of normalization is to control the numerical range of input or output data. The normalization factor can be understood as the adjustment data used in the normalization process.

[0152] S20423. Normalize the results of the first and second dimensionality reduction processes based on the normalization factor to determine the influence value of each position in the convolutional feature map on the initial local region features.

[0153] For example, when affinity is used as the influence value, the influence value can be calculated using the following formula when obtaining the normalization factor and the dimensionality reduction result in this embodiment:

[0154]

[0155] Where, r i This represents the representation vector of the i-th local region; Representing convolutional features Figure X The representation vector at the j-th position in s; ω ij This represents the influence value of position j in the i-th local region. It can be understood that position j is the index of the local region, and this index can be used to determine the influence of position j on the convolutional features. Figure X The position in s; f θ (·) represents the query transformation function, f φ (·) represents the key transformation function. Both the query transformation function and the key transformation function can be implemented as a 1×1 convolution; θ and φ represent different dimensionality reduction layers; C(Xs) represents a normalization factor, whose value can be expressed as h×w, where h is the spatial height and w is the spatial width. Where ω ijIt is the influence value representing the interaction between two pieces of information, calculated simultaneously based on the representation vector at position j in the i-th local region and the convolutional feature map.

[0156] In some feasible embodiments, such as Figure 8 As shown, the step of performing context aggregation calculation based on the influence value and the representation vectors at each position in the convolutional feature map to obtain the single-scale context region features corresponding to the convolutional feature map can include steps S20431-S20432:

[0157] S20431. When the initial local region extracted from the image to be detected is one, the influence value is multiplied by the representation vector at each position in the convolutional feature map to obtain the region feature vector at each position; the region feature vector at each position is combined to obtain the single-scale context region feature corresponding to the convolutional feature map.

[0158] The initial local region refers to the region of interest (ROI) initially extracted based on the convolutional feature map. After the image to be detected is input into the convolutional neural network, a region of interest can be determined through the region candidate network. The vector representation of the information content aggregated at each position within a single ROI is calculated, and the influence value is used as the weight of this representation vector, along with the single-scale feature map. Figure X The representation vectors at each location within the local region s are multiplied, and the summation results in the single-scale contextual ROI feature of the i-th ROI. In this embodiment, when only one local region is extracted, a single-scale contextual region feature is calculated on a convolutional feature map of one scale for that local region.

[0159] S20432. When there are multiple initial local regions extracted from the image to be detected, the influence value is multiplied by the representation vector of each local region in the convolutional feature map to obtain the region feature vector of each local region; the region feature vectors of each local region are combined to obtain the single-scale context region feature corresponding to the convolutional feature map.

[0160] In this embodiment, when multiple regions of interest are extracted, each region of interest can have its corresponding single-scale context region features calculated on a convolutional feature map of one scale. Similarly, by calculating the vector representation of the information content aggregated at each position in a single region of interest, and using the influence value as the weight of this representation vector, it is combined with the single-scale feature map. Figure X The representation vectors of each location in the local region s are multiplied together, and the results of multiplying the representation vectors of each location are accumulated. The accumulated result is used to represent the single-scale context ROI feature of the i-th ROI. Then, the single-scale context ROI features of several ROIs are combined into a vector to form the single-scale context ROI feature.

[0161] For example, after calculating each region of interest and convolutional features Figure X After determining the affinity between each location in s, single-scale contextual region of interest features are reassigned based on affinity and contextual representation. It is understandable that this can be done based on the calculated influence value ω. ij With convolutional features Figure X The representation vectors at each position of s are multiplied, and the results are accumulated to obtain the single-scale contextual region of interest (GRI) feature of the i-th GRI. The component vectors of the single-scale contextual GRI features for each GRI are obtained, resulting in a set of single-scale contextual GRI features at different scales.

[0162] (with attached diagram) Figure 9 Taking a game testing scenario as an example, the complete implementation process of the target detection method in this application is described as follows:

[0163] Step 1: Obtain the image to be detected. It is understood that this application can control the game terminal to capture a single frame of the game interface as the current image to be detected while the game is running. For example, the target detection object can connect different models of game terminals to the server. The target detection object installs the game application to be tested on the different models of game terminals through the server. After the game application is installed, the target detection object controls the game application installed on the game terminal to start. Responding to the game detection command, the server controls the game application to jump to the interface to be detected, and then controls the game terminal to take a screenshot of the interface to be detected, thus obtaining the image to be detected.

[0164] Step 2: Extract initial local region features from the game interface image to be detected, and extract global context information from the game interface image to be detected. After obtaining the image to be detected, the embodiment can input the image to be detected into a convolutional neural network, and output the global context information and global image features of the image to be detected. Then, a region candidate network is used to determine local regions on the image to be detected as regions of interest, and the features of the global image features belonging to the local region of interest are used as initial local region features. It can be understood that the region candidate network uses bounding boxes to mark the areas where one or more target game objects are located on the image to be detected as regions of interest. In the embodiment, when the convolutional neural network extracts global context information, a preset downsampling rate can be combined with a deep residual network in the convolutional neural network to extract visual features from the image to be detected, and the channel dimension, spatial height, and spatial width are determined through the visual extraction process. Then, a visual feature map can be obtained by determining the channel dimension, spatial height, and spatial width, and then the global context information in the surrounding environment image is determined based on the pixels at each position on the visual feature map.

[0165] Step 3: Obtain multi-scale context information of the image to be detected based on global context information. After obtaining the global context information, it can be input into the spatial context representation module, which outputs multiple single-scale context information at different scales. It can be understood that the spatial context representation module obtains context information from a single frame of the game interface at different scales, thus obtaining multiple single-scale context information at different scales. For example, during the processing of the spatial context representation module, the scale size corresponding to each single-scale context information in the pyramid can be configured first, and then the regions corresponding to the scale size of the global context information are pooled and aggregated according to each scale size. It can be understood that the pooling processing in the embodiment includes max pooling or average pooling. After completing the pooling aggregation at each scale size, multiple single-scale context information can be obtained. These multiple single-scale context information are then concatenated to obtain multi-scale context information.

[0166] Step 4: Based on each single-scale contextual information and the initial local region features, determine multiple single-scale contextual region features at different scales. Concatenate these single-scale contextual region features to obtain multi-scale contextual region features. Concatenate the multi-scale contextual region features with the initial local region features to obtain the target region features. In this embodiment, multiple single-scale contextual information and the initial local region features at different scales can be input into the spatial context dependency module to obtain multi-scale contextual region features. The spatial context dependency module calculates the influence values ​​of each position on the convolutional feature map with the initial local region features and combines this with the representation vectors at each position in the convolutional feature map to determine multiple single-scale contextual region features at different scales. Then, the multiple single-scale contextual region features at different scales are concatenated with the initial local region features using the concat() function. This provides features outside the local region through features at multiple different scales, thereby obtaining more accurate target region features. For example, the calculation process of the influence values ​​is as follows:

[0167]

[0168] Where, r i This represents the representation vector of the i-th local region; Representing convolutional features Figure X The representation vector at the j-th position in s; ω ij f represents the influence value of position j in the i-th local region; θ (·) represents the query transformation function, f φ (·) denotes the bond transformation function; θ and φ both represent different dimensionality reduction layers; C(Xs) represents a normalization factor, the value of which can be expressed as h×w, where h is the spatial height and w is the spatial width.

[0169] Step 5: Identify the features of the target region to obtain the target category information corresponding to each target region in the image to be detected. After obtaining the target region features, the embodiment uses a fully connected layer to identify the category of the target object within the target region. It is understood that after obtaining the category of the target object, the server can control the display terminal to show that category information.

[0170] For example, in a game testing scenario, when the game terminal will... Figure 10 After a single frame of the target game interface is sent to the server, the server can then run... Figure 2 The target detection method shown detects the target type. From... Figure 10 It is known that there are several target objects on the single frame image 1010 of the target game interface. When there are partially overlapping target objects, this embodiment can also accurately identify the overlapping target objects 1020 based on the multi-scale context region features determined by the global context information, and mark the target objects on the single frame image 1010 of the target game interface with a box 1030.

[0171] like Figure 11 As shown, a detection model training method includes steps R1101-R1106:

[0172] R1101, Obtain the image training set;

[0173] The image training set can include positive samples and negative samples. Positive samples are the samples corresponding to the categories that the model needs to correctly predict or classify; negative samples can be a set of samples constructed from any sample data that does not belong to the positive samples. It can be understood that historical images corresponding to the scene to be detected can be used to form the image training sample set. In this training sample set, each target in each training sample corresponds to the correct category information.

[0174] R1102: Extract initial local region features from each sample image in the image training set, and extract global context information from each sample image;

[0175] In this context, a local region can be understood as a region of interest within a sample image. Local region features refer to the image features within that region of interest. Contextual information can include semantic contextual information, spatial contextual information, and scale contextual information. Semantic and spatial contextual information can both be understood as interaction information between different objects and interaction information between objects and the scene. Global contextual information includes interaction information between different objects in the sample image and interaction information between objects and the background.

[0176] R1103. Obtain multi-scale context information of sample images based on global context information; wherein, multi-scale context information includes single-scale context information at multiple different scales;

[0177] In this context, image scale refers to the coarseness or fineness of the image content. Scale is used to simulate the distance of a target object from another object. Multi-scale refers to different spatial sizes. This embodiment extracts global contextual information from the image to be detected at different scales to obtain contextual information at different scales. For example, taking the detection of walking objects on a road as an example, different scales include 1.0, 3.0, and 5.0. The single-scale contextual information corresponding to the road sample image at scale 1.0, scale 3.0, and scale 5.0 is extracted respectively. The single-scale contextual information corresponding to scales 1.0, 3.0, and 5.0 is combined into multi-scale contextual information, thereby extracting contextual information from different scales.

[0178] R1104. Based on each single-scale context information and the initial local region features, determine multiple single-scale context region features at different scales; connect the single-scale context region features to obtain multi-scale context region features, and connect the multi-scale context region features with the initial local region features to obtain target region features.

[0179] The connection can be achieved using the `concat()` function to join two or more pieces of information. Taking a game testing scenario as an example, after obtaining multiple single-scale contextual information and initial local region features of the game interface sample image, this embodiment can generate multiple different-scale contextual information corresponding to the game interface sample image based on the relationship between these two pieces of information. Then, the `concat()` function is used to connect the various single-scale contextual region features into multi-scale contextual region features, thereby providing features beyond the initial local region features during model training.

[0180] R1105. Identify the features of the target region to obtain the predicted category information corresponding to each target region in the sample image;

[0181] In this embodiment, the target region features corresponding to the game interface sample image can be input into a fully connected layer to obtain the target region features for classification and prediction, thereby obtaining the predicted category information corresponding to all target regions in the game interface sample image.

[0182] R1106. Based on the predicted category information and the correct category information of each target region in the sample image, calculate the loss value of the predicted category information, and correct the parameters of the detection model based on the loss value.

[0183] In this embodiment, the loss value of category information can be calculated using Softmax and cross-entropy. Then, by differentiating the loss function and employing optimization operators such as gradient descent, the model hyperparameters are calculated to correct the parameters of the recommendation model, ultimately training the object detection model. For example, the parameters used to correct the object detection model in this embodiment may include: hyperparameters of the convolutional neural network, parameters of the spatial context representation module, parameters of the spatial context dependency module, hyperparameters of the attention mechanism, etc.

[0184] Refer to the instruction manual. Figure 11 Taking a game scenario as an example, the complete process of training the target detection model in this embodiment of the invention is described as follows:

[0185] In the backend of a game server, this embodiment first acquires historical data of single-frame image samples of the game interface corresponding to the current game, and uses single-frame image samples of game interfaces with target objects as positive examples and single-frame image samples of game interfaces without target objects as negative examples. Then, through the global context information of each game interface's single-frame image sample, information beyond local region features is provided to the model during object detection training, thereby improving the model training accuracy. Furthermore, based on the global context information, single-scale context information at multiple different scales in the game interface's single-frame image samples is obtained, which can improve the model training accuracy when training for small object detection. Finally, by differentiating the loss function and using optimization operators such as gradient descent, the model's hyperparameters are calculated to better optimize the model.

[0186] This invention also discloses another target detection method, which includes steps T001-T002:

[0187] T001. In response to the detection command, acquire the image to be detected and send the image to be detected to the target server so that the target server can perform target detection on the image to be detected and identify the target category information corresponding to each target region in the image to be detected.

[0188] In this embodiment, after receiving the target instruction from the server, the image acquisition terminal can acquire the image to be detected using a screenshot tool or a photo-taking tool, and can send the image to be detected to the server via wireless transmission. After completing the target category detection of the image to be detected, the server returns the detected target category information to the image acquisition terminal.

[0189] T002. Receive the target category information identified by the target server and display the target detection results;

[0190] In this embodiment, after receiving target category information from the server, the image acquisition terminal can display the target category information on the object interaction interface of the image acquisition terminal.

[0191] Taking vehicle type detection on the road as an example, the process of determining the target category information is as follows:

[0192] This embodiment extracts global context information and initial local region features of the road street view image sent by the terminal. Then, it obtains single-scale context information of the road street view image at multiple different scales through the global context information. Next, it combines the initial local region features to determine single-scale context region features at multiple different scales, and then performs target type recognition on the road street view image based on the single-scale context region features at multiple different scales.

[0193] Combination Figure 12 Taking a game testing scenario as an example, the complete process of the target detection method in this embodiment of the invention is described as follows:

[0194] The target device places different models of mobile game terminals in a test room and connects them to a PC (Personal Computer) with data processing capabilities via cables. The target device installs the game application to be tested on each mobile game terminal via the PC. After installation, the PC launches the game application. Responding to the detection command sent by the PC, the mobile game terminal simultaneously navigates to the test interface. The PC then takes a screenshot of each frame of the test interface, obtaining a screenshot of the game interface for each frame as the image to be tested. After obtaining the game interface screenshot, the PC identifies the global context information corresponding to the screenshot and obtains single-scale context region features at multiple different scales based on this global context information. This provides multiple features at different scales when performing target object recognition on the game interface screenshot, thereby improving the accuracy of the target object recognition results.

[0195] It should be noted that any of the target detection methods provided in this application can be applied not only to game scenarios but also to other technical fields, such as vehicle detection, face detection, and animal and plant detection. The specific implementation process in this specification is merely an example to illustrate the feasibility of the method and does not limit the application scenarios of the method.

[0196] This invention also discloses a target detection device, which includes:

[0197] The first module is used to acquire the image to be detected;

[0198] The second module is used to extract initial local region features from the image to be detected and to extract global context information from the image to be detected.

[0199] The third module is used to obtain multi-scale context information of the image to be detected based on global context information; wherein, multi-scale context information includes single-scale context information at multiple different scales;

[0200] The fourth module is used to determine single-scale context region features at multiple different scales based on each single-scale context information and the initial local region features;

[0201] The fifth module is used to connect the features of each single-scale context region to obtain multi-scale context region features, and to connect the multi-scale context region features with the initial local region features to obtain the target region features.

[0202] The sixth module is used to identify the features of the target region and obtain the target category information corresponding to each target region in the image to be detected.

[0203] In a game scenario, the target detection device of this embodiment extracts global context information and local region features from the screenshot of the game interface to be detected. When performing target detection, it can use global context information to provide other global information besides the local region for target detection, thereby improving the accuracy of the target detection results. Furthermore, the target detection device of this embodiment obtains multiple single-scale context region features at different scales based on the global context information. Then, it can combine the single-scale context region features at multiple different scales with the initial local region features to obtain the target region features. When performing target category recognition on the target region, it can use features at different spatial scales to perform target detection, thereby improving the accuracy of small target detection.

[0204] This invention also discloses another target detection device, comprising:

[0205] The seventh module is used to respond to the detection command, acquire the image to be detected, and send the image to be detected to the target server so that the target server can perform target detection on the image to be detected and identify the target category information corresponding to each target region in the image to be detected.

[0206] The eighth module is used to receive target category information identified by the target server and display the target detection results; wherein, the target category information is based on Figure 2 The target detection method shown is determined.

[0207] In a target object detection scenario, the shooting terminal is designated as the seventh module, and the server as the eighth module. In this embodiment, the shooting terminal can send the captured image of the target object to the server. The server then extracts the global context information and local region features of the image. Based on the global context information, multiple single-scale context region features at different scales are obtained. These single-scale context region features are combined with the initial local region features to obtain the target region features. This allows for accurate target category detection information when identifying the target region, utilizing features at different spatial scales. The target category detection information is then returned to the shooting terminal for display.

[0208] This invention also discloses a detection model training device, comprising:

[0209] The ninth module is used to acquire the image training set;

[0210] The tenth module is used to extract initial local region features from each sample image in the image training set and to extract global context information from each sample image.

[0211] The eleventh module is used to obtain multi-scale context information of sample images based on global context information; wherein, multi-scale context information includes single-scale context information at multiple different scales;

[0212] The twelfth module is used to determine single-scale context region features at multiple different scales based on each single-scale context information and the initial local region features;

[0213] The thirteenth module is used to connect the features of each single-scale context region to obtain multi-scale context region features, and to connect the multi-scale context region features with the initial local region features to obtain the target region features.

[0214] The fourteenth module is used to identify the features of the target region and obtain the predicted category information corresponding to each target region in the sample image;

[0215] The fifteenth module is used to calculate the loss value of the predicted category information based on the predicted category information and the correct category information of each target region in the sample image, and to correct the parameters of the detection model based on the loss value.

[0216] In the road vehicle detection scenario, this embodiment acquires historical data of street view image samples, using street view image samples containing target vehicles as positive examples and those without target vehicles as negative examples. Then, by utilizing the global context information of each street view image sample, information beyond local region features is provided to the model during target detection training, thereby improving model training accuracy. Furthermore, single-scale context information at multiple different scales within the street view image samples is obtained based on the global context information, which can improve model training accuracy when training for small target detection. Finally, by differentiating the loss function and employing optimization operators such as gradient descent, the model's hyperparameters are calculated to better optimize the model.

[0217] It should be noted that any of the target detection devices provided in this application can be applied not only to game scenarios but also to other technical fields, such as vehicle detection, face detection, and animal and plant detection. The specific implementation process in this specification is merely an example of feasible methods of the device and does not limit the application scenarios of the device.

[0218] This invention also provides an electronic device, which includes a processor and a memory; the memory stores a program; the processor executes the program to perform the aforementioned target detection method; the electronic device has the function of carrying and running the business data processing software system provided in this invention, such as a personal computer (PC), mobile phone, smartphone, personal digital assistant (PDA), wearable device, handheld computer (PPC), tablet computer, vehicle terminal, etc.

[0219] For example, when a game player logs into the game client through any electronic device, the embodiment takes a screenshot of the game interface through the electronic device, and the processor performs target detection on the screenshot of the game interface. It can be understood that after obtaining the screenshot of the game interface, the processor extracts the global context information and local region features of the screenshot of the game interface, so as to provide other global information besides the local region through the global context information for target detection, thereby improving the accuracy of the target detection results; and, based on the global context information, it obtains single-scale context region features at multiple different scales, so that when performing target category recognition on the target region, features at different spatial scales can be used for target detection, thereby improving the accuracy of small target detection.

[0220] This invention also provides a computer-readable storage medium storing a program that is executed by a processor to implement the aforementioned target detection method. Simultaneously, this invention also provides a computer program product or computer program comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned virtual target detection method.

[0221] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0222] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0223] From the embodiments provided in the above specification, it is clear that the technical solution of this application has at least the following beneficial effects:

[0224] The technical solution of this invention can provide local information beyond local region features for the target detection process through the global context information of the image to be detected, thereby improving the accuracy of target detection. Furthermore, this invention also acquires single-scale context region features at multiple different scales, thereby performing target recognition through features at different spatial scales. When the target to be detected is a small target or the color difference between the target and the background in the detected image is not obvious, the single-scale context region features at multiple different scales of this invention can more accurately detect small targets or targets with indistinct colors, thus improving the accuracy of target detection.

[0225] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0226] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0227] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0228] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0229] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0230] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0231] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0232] The above is a detailed description of the preferred embodiments of the present invention, but the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A target detection method, characterized in that, include: Acquire the image to be detected; Initial local region features are extracted from the image to be detected, and global context information is extracted from the image to be detected. The multi-scale context information of the image to be detected is obtained based on the global context information; wherein, the multi-scale context information includes single-scale context information at multiple different scales; Determine the convolutional feature map corresponding to each of the single-scale context information; Calculate the influence value of each position in the convolutional feature map on the initial local region features; When the initial local region extracted from the image to be detected is one, the influence value is multiplied by the representation vector of each position in the convolutional feature map to obtain the region feature vector of each position; the region feature vector of each position is combined to obtain the single-scale context region feature corresponding to the convolutional feature map. When there are multiple initial local regions extracted from the image to be detected, the influence value is multiplied by the representation vector of each local region in the convolutional feature map to obtain the region feature vector of each local region; the region feature vectors of each local region are combined to obtain the single-scale context region feature corresponding to the convolutional feature map. The single-scale context region features are concatenated to obtain multi-scale context region features, and the multi-scale context region features are concatenated with the initial local region features to obtain target region features. The target region features are identified to obtain the target category information corresponding to each target region in the image to be detected.

2. The target detection method according to claim 1, characterized in that, The step of extracting global context information from the image to be detected includes: Based on a preset downsampling rate, visual features are extracted from the image to be detected using a deep residual network; Based on the channel dimension, spatial height, and spatial width determined by the visual feature extraction, the convolutional feature map output by the deep residual network is obtained. The global context information is determined based on the convolutional feature map.

3. A target detection method according to claim 1 or 2, characterized in that, The step of obtaining the multi-scale context information of the image to be detected based on the global context information includes: Configure the scale size corresponding to each single-scale context information; Based on each scale size, the regions in the global context information corresponding to the current scale size are pooled and aggregated to obtain the single-scale context information corresponding to the current scale size; The multi-scale context information is determined based on the single-scale context information corresponding to all scale sizes.

4. The target detection method according to claim 3, characterized in that, The step of pooling and aggregating the regions corresponding to the current scale size in the global context information according to each scale size to obtain the single-scale context information corresponding to the current scale size includes: Sequentially obtain the size of each scale as the current scale size; Based on the current scale, determine the region to be pooled from the global context information; By performing max pooling or average pooling, the region to be pooled is aggregated to obtain the single-scale context information corresponding to the current scale size. In this context, the information at each location in the single-scale context information represents the information of all locations in the region to be pooled.

5. The target detection method according to claim 1, characterized in that, The calculation of the influence value of each position in the convolutional feature map on the initial local region features includes: The representation vectors at each position in the convolutional feature map are subjected to a first dimensionality reduction process, and the representation vectors of the initial local region features are subjected to a second dimensionality reduction process. Based on the spatial height and spatial width of the convolutional feature map, a normalization factor is constructed; The results of the first dimensionality reduction process and the second dimensionality reduction process are normalized according to the normalization factor to determine the influence value of each position in the convolutional feature map on the initial local region features.

6. A target detection method, characterized in that, include: In response to a detection command, an image to be detected is acquired and sent to a target server, so that the target server performs target detection on the image to be detected and identifies the target category information corresponding to each target region in the image to be detected. Receive the target category information identified by the target server and display the target detection results; The target category information is determined according to the target detection method as described in any one of claims 1-5.

7. A method for training a detection model, characterized in that, include: Obtain the image training set; Initial local region features are extracted from each sample image in the image training set, and global context information is extracted from each sample image. The multi-scale context information of the sample image is obtained based on the global context information; wherein, the multi-scale context information includes single-scale context information at multiple different scales; Determine the convolutional feature map corresponding to each of the single-scale context information; Calculate the influence value of each position in the convolutional feature map on the initial local region features; When the initial local region extracted from the sample image is one, the influence value is multiplied by the representation vector of each position in the convolutional feature map to obtain the region feature vector of each position; the region feature vector of each position is combined to obtain the single-scale context region feature corresponding to the convolutional feature map. When there are multiple initial local regions extracted from the sample image, the influence value is multiplied by the representation vector of each local region in the convolutional feature map to obtain the region feature vector of each local region; the region feature vectors of each local region are combined to obtain the single-scale context region feature corresponding to the convolutional feature map. The single-scale context region features are concatenated to obtain multi-scale context region features, and the multi-scale context region features are concatenated with the initial local region features to obtain target region features. The features of the target region are identified to obtain the predicted category information corresponding to each target region in the sample image; Based on the predicted category information and the correct category information of each target region in the sample image, the loss value of the predicted category information is calculated, and the parameters of the detection model are corrected based on the loss value.

8. A target detection device, characterized in that, include: The first module is used to acquire the image to be detected; The second module is used to extract initial local region features from the image to be detected and to extract global context information from the image to be detected. The third module is used to obtain multi-scale context information of the image to be detected based on the global context information; wherein, the multi-scale context information includes single-scale context information at multiple different scales; The fourth module is used to determine the convolutional feature map corresponding to each of the single-scale context information. Calculate the influence value of each position in the convolutional feature map on the initial local region features; When the initial local region extracted from the image to be detected is one, the influence value is multiplied by the representation vector of each position in the convolutional feature map to obtain the region feature vector of each position; the region feature vector of each position is combined to obtain the single-scale context region feature corresponding to the convolutional feature map. When there are multiple initial local regions extracted from the image to be detected, the influence value is multiplied by the representation vector of each local region in the convolutional feature map to obtain the region feature vector of each local region; the region feature vectors of each local region are combined to obtain the single-scale context region feature corresponding to the convolutional feature map. The fifth module is used to connect the single-scale context region features to obtain multi-scale context region features, and connect the multi-scale context region features with the initial local region features to obtain target region features; The sixth module is used to identify the features of the target region and obtain the target category information corresponding to each target region in the image to be detected.

9. The target detection device according to claim 8, characterized in that, The second module is used for: Based on a preset downsampling rate, visual features are extracted from the image to be detected using a deep residual network; Based on the channel dimension, spatial height, and spatial width determined by the visual feature extraction, the convolutional feature map output by the deep residual network is obtained. The global context information is determined based on the convolutional feature map.

10. The target detection device according to claim 8 or 9, wherein the third module is configured to: Configure the scale size corresponding to each single-scale context information; Based on each scale size, the regions in the global context information corresponding to the current scale size are pooled and aggregated to obtain the single-scale context information corresponding to the current scale size; The multi-scale context information is determined based on the single-scale context information corresponding to all scale sizes.

11. A target detection device, characterized in that, include: The seventh module is used to respond to a detection command, acquire an image to be detected, and send the image to be detected to the target server, so that the target server performs target detection on the image to be detected and identifies the target category information corresponding to each target region in the image to be detected; The eighth module is used to receive the target category information identified by the target server and display the target detection results; The target category information is determined according to the target detection method as described in claims 1-5.

12. A detection model training device, characterized in that, include: The ninth module is used to acquire the image training set; The tenth module is used to extract initial local region features from each sample image of the image training set, and to extract global context information from each sample image; The eleventh module is used to obtain multi-scale context information of the sample image based on the global context information; wherein, the multi-scale context information includes single-scale context information at multiple different scales; The twelfth module is used to determine the convolutional feature map corresponding to each of the single-scale context information; Calculate the influence value of each position in the convolutional feature map on the initial local region features; When the initial local region extracted from the sample image is one, the influence value is multiplied by the representation vector of each position in the convolutional feature map to obtain the region feature vector of each position; the region feature vector of each position is combined to obtain the single-scale context region feature corresponding to the convolutional feature map. When there are multiple initial local regions extracted from the sample image, the influence value is multiplied by the representation vector of each local region in the convolutional feature map to obtain the region feature vector of each local region; the region feature vectors of each local region are combined to obtain the single-scale context region feature corresponding to the convolutional feature map. The thirteenth module is used to connect the single-scale context region features to obtain multi-scale context region features, and connect the multi-scale context region features with the initial local region features to obtain target region features. The fourteenth module is used to identify the features of the target region and obtain the predicted category information corresponding to each target region in the sample image; The fifteenth module is used to calculate the loss value of the predicted category information based on the predicted category information and the correct category information of each target region in the sample image, and to correct the parameters of the detection model based on the loss value.

13. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the target detection method as described in any one of claims 1 to 6 or the detection model training method as described in claim 7.

14. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the target detection method as described in any one of claims 1 to 6 or the detection model training method as described in claim 7.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the target detection method as described in any one of claims 1 to 6 or the detection model training method as described in claim 7.

Images