An image recognition method, apparatus, device, and readable storage medium

Abstract

This application discloses an image recognition method, apparatus, device, and readable storage medium. The method includes: acquiring a target image; inputting the target image into an image recognition model; extracting image features of the target image in the image recognition model; inputting the image features into an image classification component and a location prediction component in the image recognition model; in the image classification component, performing convolution processing on the image features using a first convolution parameter to obtain a classification convolution feature map; in the location prediction component, performing convolution processing on the image features using a second convolution parameter to obtain a location convolution feature map; outputting the object category of the target object in the target image based on the classification convolution feature map; and outputting the location information of the target object in the target image based on the location convolution feature map. Using this application can improve image recognition efficiency.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image recognition method, apparatus, device, and readable storage medium. Background Technology

[0002] Currently, classifying and selecting bounding boxes for objects within an image requires different recognition networks. For object classification, a classification network can be used; for bounding box selection, an object detection network can be used.

[0003] In existing technologies, methods for classifying objects and selecting bounding boxes using two different recognition networks require extracting image features from each network. These features are then convolved separately by the two networks to obtain the object classification and predicted object boundaries. In other words, extracting image features from the same image requires two or more computational steps, leading to a significant computational burden and impacting recognition efficiency. Summary of the Invention

[0004] This application provides an image recognition method, apparatus, device, and readable storage medium, which can improve image recognition efficiency.

[0005] One embodiment of this application provides an image recognition method, including:

[0006] Acquire the target image, input the target image into the image recognition model, and extract the image features of the target image in the image recognition model;

[0007] Image features are input into the image classification component and the location prediction component in the image recognition model, respectively; the image classification component contains a first convolutional parameter that focuses on object classification, and the location prediction component contains a second convolutional parameter that focuses on location information prediction.

[0008] In the image classification component, the image features are convolved using the first convolution parameter to obtain a classification convolution feature map.

[0009] In the location prediction component, the image features are convolved using the second convolution parameter to obtain the location convolution feature map.

[0010] Based on the classification convolutional feature map, the object category of the target object in the target image is output, and based on the position convolutional feature map, the position information of the target object in the target image is output.

[0011] One embodiment of this application provides an image recognition device, including:

[0012] The image acquisition module is used to acquire the target image and input the target image into the image recognition model.

[0013] The feature extraction module is used to extract image features from the target image in the image recognition model;

[0014] The feature input module is used to input image features into the image classification component and the location prediction component in the image recognition model, respectively; the image classification component contains a first convolutional parameter that focuses on object classification, and the location prediction component contains a second convolutional parameter that focuses on location information prediction.

[0015] The first feature convolution module is used in the image classification component to perform convolution processing on image features through the first convolution parameters to obtain a classification convolution feature map;

[0016] The second feature convolution module is used in the location prediction component to perform convolution processing on image features through the second convolution parameters to obtain a location convolution feature map.

[0017] The category output module is used to output the object category of the target object in the target image based on the classification convolution feature map;

[0018] The position output module is used to output the position information of the target object in the target image based on the position convolution feature map.

[0019] The target image consists of K grid pixels after image division. The image recognition model includes a residual network, which comprises a first feature extraction unit, a second feature extraction unit, a third feature extraction unit, and a convolutional layer. K is an integer greater than 1.

[0020] The feature extraction module includes:

[0021] The first pixel feature output unit is used to input the target image into the first feature extraction unit, and extract the pixel features of each grid pixel in the target image in the first feature extraction unit to obtain K first pixel features;

[0022] The second pixel feature output unit is used to input K first pixel features into the second feature extraction unit, and to perform convolution processing on each first pixel feature in the second feature extraction unit to obtain K second pixel features.

[0023] The third pixel feature output unit is used to input K second pixel features into the third feature extraction unit. In the third feature extraction unit, each second pixel feature is convolved to obtain K third pixel features.

[0024] The target pixel feature selection unit is used to select K target pixel features from K first pixel features, K second pixel features, and K third pixel features;

[0025] The feature convolution unit is used to input K target pixel features into the convolutional layer, and perform convolution processing on the K target pixel features in the convolutional layer to obtain convolutional pixel features;

[0026] The image feature generation unit is used to generate image features of the target image based on the convolution pixel features.

[0027] The device also includes:

[0028] The scaling module is used to acquire the original image and obtain the image scaling ratio;

[0029] The image scaling module is used to scale the original image according to the image scaling ratio to obtain a transition image;

[0030] The image segmentation module is used to segment the transition image to obtain a target image consisting of K grid pixels.

[0031] Among them, the K target pixel features are K third pixel features; the convolutional layer includes the first convolutional layer;

[0032] Feature convolutional units include:

[0033] The first convolutional subunit is used to input K third pixel features into the first convolutional layer, and to perform convolution processing on each third pixel feature in the first convolutional layer to obtain K initial convolutional pixel features.

[0034] The first downsampling subunit is used to obtain the first downsampling factor corresponding to the third feature extraction unit, and to downsample the K initial convolutional pixel features according to the first downsampling factor to obtain the convolutional pixel features.

[0035] The convolutional layer includes a first convolutional layer and a second convolutional layer;

[0036] Feature convolutional units include:

[0037] The second convolutional subunit is used to input K third pixel features into the first convolutional layer, and to perform convolution processing on each third pixel feature in the first convolutional layer to obtain K initial convolutional pixel features.

[0038] The second downsampling subunit is used to obtain the first downsampling factor corresponding to the third feature extraction unit, and downsamples the K initial convolutional pixel features according to the first downsampling factor to obtain the first downsampled pixel features.

[0039] The third downsampling subunit is used to obtain the second downsampling factor corresponding to the second feature extraction unit, and downsamples K second pixel features according to the second downsampling factor to obtain the second downsampled pixel features.

[0040] The upsampling subunit is used to determine the upsampling factor of the first image based on the first downsampling factor and the second downsampling factor, and to upsample the first downsampled pixel features based on the first image upsampling factor to obtain the upsampled pixel features;

[0041] The feature concatenation subunit is used to concatenate the second downsampled pixel features with the upsampled pixel features if the K target pixel features are K second pixel features, to obtain the first concatenated feature. The first concatenated feature is then input into the second convolutional layer, where it is convolved to obtain the convolutional pixel feature.

[0042] The convolutional layer also includes a third convolutional layer;

[0043] The device also includes:

[0044] The first feature concatenation module is used to concatenate the second downsampled pixel features with the upsampled pixel features if the K target pixel features are K first pixel features, to obtain the first concatenated feature, and input the first concatenated feature into the second convolutional layer, and perform convolution processing on the first concatenated feature in the second convolutional layer to obtain the convolutional concatenated feature.

[0045] The feature downsampling module is used to obtain the third downsampling factor corresponding to the first feature extraction unit, and downsample the K first pixel features according to the third downsampling factor to obtain the third downsampled pixel features.

[0046] The feature upsampling module is used to determine the upsampling factor of the second image based on the third downsampling factor and the second downsampling factor, and upsample the convolutional concatenation features according to the upsampling factor of the second image to obtain the upsampled concatenation features;

[0047] The second feature concatenation module is used to concatenate the third downsampled pixel features with the upsampled concatenation features to obtain the second concatenated features. The second concatenated features are then input into the third convolutional layer, where they are convolved to obtain the convolutional pixel features.

[0048] The category output module includes:

[0049] The existence probability acquisition unit is used to acquire classification feature points in the classification convolution feature map and to acquire the existence probability of the object corresponding to the classification feature point. The object existence probability refers to the probability that the target object exists in the prediction box to which the classification feature point belongs. The prediction box is used to predict the position of the target object in the target image.

[0050] The prediction probability acquisition unit is used to acquire the predicted probability of the category corresponding to the classification feature point;

[0051] The category determination unit is used to obtain the maximum category prediction probability from the category prediction probability if the probability of an object is greater than the probability threshold, and determine the category corresponding to the maximum category prediction probability as the object category predicted by the classification feature point.

[0052] The aforementioned category determination unit is also used to determine the object category predicted by the classification feature point as the object category of the prediction box to which the classification feature point belongs.

[0053] The position output module includes:

[0054] The position parameter acquisition unit is used to acquire position feature points in the position convolution feature map and to acquire the predicted position parameters corresponding to the position feature points.

[0055] The pixel block acquisition unit is used to acquire the grid pixel block corresponding to the position feature point in K grid pixels; the position feature point is obtained by convolution of the grid pixel block, and the size of the grid pixel block is determined by the downsampling factor corresponding to the K target pixel features;

[0056] The center coordinate acquisition unit is used to obtain the coordinates of the center position of the grid pixel block;

[0057] The location information determination unit is used to determine the location information of the target object based on the predicted location parameters and the center location coordinates.

[0058] The predicted position parameters include the position offset, the width of the predicted bounding box, and the height of the predicted bounding box; the predicted bounding box is used to predict the position of the target object in the target image.

[0059] Location information determination unit, including:

[0060] The center position determination sub-unit is used to determine the center position information corresponding to the prediction box based on the position offset and the center position coordinates;

[0061] The location information determination subunit is used to determine the location information of the prediction box in the target image based on the center location information, the width of the prediction box, and the height of the prediction box, and then determine the location information of the prediction box in the target image as the location information of the target object.

[0062] The device also includes:

[0063] The sample acquisition module is used to acquire image samples, input the image samples into the sample image recognition model, and extract the sample features of the image samples in the sample image recognition model;

[0064] The sample input module is used to input sample features into the sample image classification component and the sample location prediction component of the sample image recognition model, respectively; the sample image classification component contains a first sample convolution parameter that focuses on object classification, and the sample location prediction component contains a second sample convolution parameter that focuses on location information prediction.

[0065] The classification convolution module is used to perform convolution processing on sample features through the first sample convolution parameters in the sample image classification component to obtain the sample classification convolution feature map.

[0066] The location convolution module is used in the sample location prediction component to perform convolution processing on the sample features through the second sample convolution parameters to obtain the sample location convolution feature map.

[0067] The result determination module is used to determine the predicted object category of the sample object in the image sample based on the sample classification convolutional feature map, and to determine the predicted location information of the sample object based on the sample location convolutional feature map.

[0068] The loss value determination module is used to obtain the object category label of the sample object, obtain the location label information of the sample object, and determine the total loss value based on the predicted object category, object category label, predicted location information, and location label information.

[0069] The parameter adjustment module is used to adjust the first sample convolution parameters and the second sample convolution parameters according to the total loss value, so as to obtain the first convolution parameters corresponding to the first sample convolution parameters and the second convolution parameters corresponding to the second sample convolution parameters.

[0070] The model determination module is used to determine the sample image classification component including the first convolution parameter as the image classification component, the sample location prediction component including the second convolution parameter as the location prediction component, and the sample image recognition model including the image classification component and the location prediction component as the image recognition model.

[0071] The loss value determination module includes:

[0072] The classification loss determination unit is used to determine the classification loss value based on the predicted object category and the object category label;

[0073] The location loss determination unit is used to determine the location loss value based on the location label information and the predicted location information.

[0074] The loss value generation unit is used to generate the total loss value based on the classification loss value and the location loss value.

[0075] The parameter adjustment module includes:

[0076] The first derivative unit is used to determine the first partial derivative between the total loss value and the predicted object category, and to adjust the first sample convolution parameters based on the first partial derivative to obtain the first convolution parameters.

[0077] The second derivative unit is used to determine the second partial derivative between the total loss value and the predicted location information, and to adjust the second sample convolution parameters based on the second partial derivative to obtain the second convolution parameters.

[0078] This application provides a computer device, including: a processor, a memory, and a network interface;

[0079] The processor is connected to a memory and a network interface. The network interface is used to provide data communication functions, the memory is used to store computer programs, and the processor is used to call the computer programs to execute the method in one aspect of the embodiments of this application.

[0080] This application provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, perform the method in one aspect of the embodiments of this application.

[0081] One aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method provided in one aspect of the embodiments of this application.

[0082] In this embodiment, after extracting the image features of the target image, the image features are input into the image classification component and the location prediction component respectively. The image features are convolved by the first convolution parameter in the image classification component to obtain a classification convolution feature map. The object category of the target object in the target image can then be output based on the classification convolution feature map. At the same time, the image features are convolved by the second convolution parameter in the location prediction component to obtain a location convolution feature map. The location information of the target object in the target image can then be output based on the location convolution feature map. Because the first convolutional parameter in the image classification component focuses on object classification, while the second convolutional parameter in the location prediction component focuses on location information prediction, and these two parameters are independent and have different focuses, the accuracy of image classification can be improved when using the first convolutional parameter to classify image features, and the accuracy of location prediction can be improved when using the second convolutional parameter to predict image features. Furthermore, since both the image classification and location prediction components are included in the image recognition model, the image recognition model only needs to extract the image features of the target image once. By inputting these image features into the image classification and location prediction components respectively, image feature sharing can be achieved, thereby reducing computational load and improving recognition efficiency. Attached Figure Description

[0083] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0084] Figure 1 This is a network architecture diagram provided in an embodiment of this application;

[0085] Figure 2a This is a schematic diagram of a scenario provided in an embodiment of this application;

[0086] Figure 2b This is a schematic diagram illustrating the correspondence between grid pixel blocks and classification feature points provided in an embodiment of this application;

[0087] Figure 2c This is a schematic diagram illustrating the correspondence between grid pixel blocks and positional feature points provided in an embodiment of this application;

[0088] Figure 2d This is an application scenario diagram provided in an embodiment of this application;

[0089] Figure 2e This is an application scenario diagram provided in an embodiment of this application;

[0090] Figure 3 This is a schematic flowchart of an image recognition method provided in an embodiment of this application;

[0091] Figure 4 This is a schematic diagram of an image feature extraction method provided in an embodiment of this application;

[0092] Figure 5 This is a schematic diagram illustrating the determination of classification convolutional feature maps and positional convolutional feature maps provided in an embodiment of this application;

[0093] Figure 6 This is a schematic diagram of a model training process provided in an embodiment of this application;

[0094] Figure 7 This is a schematic diagram of the structure of an image recognition device provided in an embodiment of this application;

[0095] Figure 8 This is a schematic diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0096] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0097] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0098] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0099] The solutions provided in this application belong to the fields of computer vision (CV) and machine learning (ML) within the field of artificial intelligence.

[0100] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in tasks such as target recognition, tracking, and measurement, and further performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0101] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0102] Please see Figure 1 , Figure 1 This is a network architecture diagram provided in an embodiment of this application. For example... Figure 1 As shown, the network architecture may include a business server 1000 and a backend server cluster, wherein the backend server cluster may include multiple backend servers, such as... Figure 1 As shown, it can specifically include backend server 100a, backend server 100b, backend server 100c, ..., backend server 100n. Figure 1 As shown, backend servers 100a, 100b, 100c, ..., 100n can respectively connect to the aforementioned business server 1000 via the network, so that each backend server can interact with the business server 1000 through the network connection, and so that the aforementioned business server 1000 can receive business data from each backend server.

[0103] like Figure 1 Each backend server shown corresponds to a user terminal and can be used to store the corresponding user terminal's business data. Each user terminal can have a target application installed. When this target application runs on each user terminal, the backend server corresponding to each user terminal can store the application's business data and connect it to the aforementioned... Figure 1 The business servers 1000 shown interact with each other. The target application can include applications capable of displaying text, images, audio, and video data. For example, the application can be an image recognition application, such as a watermark recognition application, which can be used to allow users to upload images and view sensitive watermarks or violation markers; the application can also be an image classification application, which can be used to allow users to upload images and view the object category (e.g., elephant, tiger, lark, etc.) of objects contained in the image; the application can also be a video detection application, which can be used to allow users to upload videos and detect whether there are violation markers or sensitive watermarks in the video frames. The business server 1000 in this application can collect business data from the backend of these applications (such as the aforementioned backend server cluster), such as user-uploaded images and videos. Based on the collected business data, the business server 1000 can determine the object category of the objects contained in the business data, as well as the location information of the objects in the business data (images or videos). Furthermore, the business server 1000 can send the object category and location information to the backend server. The user can then use the user terminal corresponding to the backend server to determine the object category of the object within the video frame of the image or video, and view the object's location information within the video frame. The user can then perform subsequent processing based on the viewed object category and location information. For example, if a user uploads video A, and the business server 1000 detects a sensitive watermark at a certain location in video frame A of video A, the business server 1000 can return the detection result to the backend server corresponding to the user's user terminal. The user can then view this detection result on the display page of the user terminal. Subsequently, the user can use the location information (at that location in video frame A of video A) and the object category (sensitive watermark) returned by the business server 1000 to delete this sensitive watermark in video A, thus ensuring the legitimacy of video A.

[0104] This application embodiment can select one user terminal from multiple user terminals as the target user terminal. The target user terminal may include: a smart terminal with display and playback data information functions, such as a smartphone, tablet computer, or desktop computer. For example, this application embodiment can... Figure 1The user terminal corresponding to the backend server 100a shown is the target user terminal. The target user terminal can integrate the above-mentioned target application. At this time, the backend server 100a corresponding to the target user terminal can interact with the business server 1000.

[0105] For example, when a user uses a target application (such as a video detection application) on a user terminal, the business server 1000 can detect and collect information about a video uploaded by the user through the backend server corresponding to the user terminal. The business server 1000 can determine whether there is a target object (such as a person, animal, or sensitive watermark) in the video. If there is a target object in the video, the business server 1000 determines the object category to which the target object belongs and the location information of the target object in the video, and returns the object category and location information to the backend server. The user can then view the object category and location information on the display page of the user terminal corresponding to the backend server and perform subsequent processing based on this object category and location information.

[0106] Optionally, and understandably, the backend server can detect and collect business data (such as images or videos) from the corresponding user terminals, and determine the object category of the objects contained in this business data, as well as the location information of the objects within the business data (images or videos). Users can view the object categories and location information determined by the backend server on the display page of the user terminal corresponding to the backend server.

[0107] It is understood that the methods provided in this application embodiment can be executed by computer equipment, including but not limited to user terminals or business servers. User terminals and business servers can be directly or indirectly connected via wired or wireless communication, and this application does not impose any limitations on this.

[0108] For easier understanding, please refer to Figure 2a , Figure 2a This is a schematic diagram of a scenario provided in an embodiment of this application. Wherein, as... Figure 2a The service server shown can be the aforementioned service server 1000, and as follows: Figure 2a The terminal A shown can be the one described above. Figure 1 Any user terminal selected in the user terminal cluster of the corresponding embodiment, for example, the user terminal can be the aforementioned user terminal 100a.

[0109] like Figure 2aAs shown, user A uploads image 20a through user terminal A. User terminal A can send image 20a to the service server. After receiving image 20a, the service server can scale and segment the image. Specifically, the service server can obtain the image scaling ratio. Based on the scaling ratio, the service server can scale image 20a to obtain a transition image. Subsequently, the service server can segment the transition image to obtain a target image comprising K grid pixels. The image scaling ratio can be a predefined value, such as 416*416, 318*318, 256*256, etc. This application does not limit the specific form of the image scaling ratio, and examples will not be provided here. K is an integer greater than 1, and the value of K is related to the image scaling ratio. For example, if the image scaling ratio is 416*416, then K can be 416*416 = 173056. Figure 2a As shown, image 20a has dimensions of 16*8 (length 16, width 8). The business server obtains an image with a scaling ratio of 8*8 (length 8, width 8). Therefore, the business server needs to scale image 20a to an image with a size of 8*8. Figure 2a As shown, the business server can scale the length (16) of image 20a to 8. The scaling factor between 16 and 8 is 1 / 2. Therefore, in order to keep the shape of image 20a unchanged, the business server can scale the width (8) of image 20a to 4 (8 * 1 / 2 = 4). Then... Figure 2a As shown, after scaling the dimensions of image 20a, its size becomes 8*4, which is less than 8*8. To achieve the desired size of 8*8, the difference between the dimensions of 8*8 and 8*4 can be addressed (e.g., ...). Figure 2a The regions P and Q in the diagram are filled.

[0110] like Figure 2a As shown, after filling the difference regions P and Q, a transition image 20b is obtained, with a size of 8*8 pixels. The business server can then perform image segmentation on this transition image 20b. As shown, since the image scaling ratio is 8*8, the business server can divide the transition image 20b into 64 (8*8 = 64) grid pixels, where each grid pixel has the same size. This yields the target image 20c (comprising 64 grid pixels). The business server can then input this target image, comprising 64 grid pixels, into the image recognition model.

[0111] Furthermore, in this image recognition model, image features of the target image 20c can be extracted, and these image features can be input into the image classification component of the image recognition model. Through the first convolution parameter in the image classification component, the image features can be convolved to obtain a classification convolution feature map. The image features can also be input into the position prediction component of the image recognition model. Through the second convolution parameter in the position prediction component, the image features can be convolved to obtain a position convolution feature map.

[0112] like Figure 2a As shown, the classification convolutional graph output by the image classification component includes classification feature points m1, m2, m3, and m4. Each classification feature point in this convolutional graph corresponds to one of the 64 grid pixels in the target image 20c. Each classification feature point is obtained by convolving its corresponding grid pixel block. The size of each grid pixel block is determined by the downsampling factor in the image recognition model. For example, if the downsampling factor is 1 / 4, the size of each grid pixel block can be 4*4. This downsampling factor can be used to downsample the 64 grid pixels, thereby changing the number of grid pixels in the target image 20c and thus changing the size of the target image 20c. For easier understanding, please refer to [link to relevant documentation]. Figure 2b , Figure 2b This is a schematic diagram illustrating the correspondence between grid pixel blocks and classification feature points provided in an embodiment of this application. For example... Figure 2b As shown, classification feature point m1 corresponds to grid pixel block g1 in target image 20c, classification feature m2 corresponds to grid pixel block g2 in target image 20c, classification feature point m3 corresponds to grid pixel block g3 in target image 20c, and classification feature point m4 corresponds to grid pixel block g4 in target image 20c.

[0113] like Figure 2b As shown, the size of grid pixel blocks g1, g2, g3, and g4 is 4*4 (comprising a total of 16 grid pixels). This means that the downsampling factor in the image recognition model is 1 / 4. With this downsampling factor of 1 / 4, the 64 grid pixels in the target image 20c are divided into 4 grid pixel blocks (grid pixel blocks g1, g2, g3, and g4). Multiple convolutions of grid pixel block g1 yield classification feature point m1, multiple convolutions of grid pixel block g2 yield classification feature point m2, multiple convolutions of grid pixel block g3 yield classification feature point m3, and multiple convolutions of grid pixel block g4 yield classification feature point m4.

[0114] like Figure 2a As shown, the location prediction component outputs a location convolutional feature map containing four location feature points: d1, d2, d3, and d4. Each location feature point in the location convolutional feature map corresponds to one of the 64 grid pixels in the aforementioned target image 20c. Each location feature point is obtained by performing multiple convolutions on its corresponding grid pixel block. The size of each grid pixel block is determined by the downsampling factor in the image recognition model. For easier understanding, please refer to [link to relevant documentation]. Figure 2c , Figure 2c This is a schematic diagram illustrating the correspondence between grid pixel blocks and positional feature points provided in an embodiment of this application. For example... Figure 2c As shown, position feature point d1 corresponds to grid pixel block g1 in target image 20c, position feature point d2 corresponds to grid pixel block g2 in target image 20c, position feature point d3 corresponds to grid pixel block g3 in target image 20c, and position feature point d4 corresponds to grid pixel block g4 in target image 20c. Figure 2c As shown, the size of grid pixel blocks g1, g2, g3, and g4 is 4*4 (comprising a total of 16 grid pixels). This means that the downsampling factor in the image recognition model is 1 / 4. With this downsampling factor of 1 / 4, the 64 grid pixels in the target image 20c are divided into 4 grid pixel blocks (grid pixel blocks g1, g2, g3, and g4). Multiple convolutions of grid pixel block g1 yield positional feature point d1, multiple convolutions of grid pixel block g2 yield positional feature point d2, multiple convolutions of grid pixel block g3 yield positional feature point d3, and multiple convolutions of grid pixel block g4 yield positional feature point d4.

[0115] Combination Figure 2b as well as Figure 2c As can be seen, in the image recognition model, the 64 grid pixels of the target image 20c can be divided into multiple grid pixel blocks by using a downsampling factor. By convolving these grid pixel blocks, the classification feature points or location feature points corresponding to each grid pixel block can be obtained, thus yielding a classification convolutional feature map or a location convolutional feature map. It should be noted that the convolution parameters used when convolving the grid pixel blocks to obtain classification feature points are different from those used when convolving the grid pixel blocks to obtain location feature points.

[0116] Combination Figure 2b as well as Figure 2c It can be seen that Figure 2aIn the model, the classification feature point m1 and the position feature point d1 correspond to the same grid pixel block g1, the classification feature point m2 and the position feature point d2 correspond to the same grid pixel block g2, the classification feature point m3 and the position feature point d3 correspond to the same grid pixel block d3, and the classification feature point m4 and the position feature point d4 correspond to the same grid pixel block g4. The output parameters for each categorical feature point are [object presence probability: category prediction probability], while the output parameters for each location feature point are [location offset, prediction box width, prediction box height]. When a categorical feature point and a location feature point correspond to the same grid pixel block, one output parameter of the categorical feature point [object presence probability: category prediction probability] corresponds to one output parameter of the location feature point [location offset, prediction box width, prediction box height]. Combining the output parameters of the categorical and location feature points, it can be determined whether the prediction box corresponding to the location feature point contains a target object. Furthermore, the category prediction probability from the categorical feature point can determine the object category within the prediction box. Simultaneously, the location information of the prediction box in the target image can be determined using the location offset, the width of the prediction box, and the height of the prediction box. For example, ... Figure 2aAs shown, the prediction boxes corresponding to the location feature point d1 include prediction boxes h1 and h2. The output parameters corresponding to the classification feature point m1 include the output parameters corresponding to the prediction boxes h1 and h2 (that is, the object existence probability and category prediction probability corresponding to the prediction boxes h1 and h2, respectively). For example, the output parameters corresponding to the classification feature point m1 are [0.10: 0.01, 0.03, 0.98] and [0.23: 0.02, 0.01, 0.94]. Among them, the output parameters [0.10: 0.01, 0.03, 0.98] correspond to the prediction box h1. The value 0.10 in the output parameters represents the object existence probability of the prediction box h1, which is 0.10. The values ​​0.01, 0.03, and 0.98 in the output parameters are all predicted category probabilities. Among them, the predicted category corresponding to the value 0.01 is elephant, the predicted category corresponding to the value 0.03 is wolf, and the predicted category corresponding to the value 0.98 is monkey. Because the probability of the object's existence (0.10) is less than the probability threshold of 0.5, the business server can determine that no object exists in the prediction box h1. Therefore, it can directly filter out the predicted category probabilities of 0.01, 0.03, and 0.98, and no longer use these probabilities to determine the object category of the object in prediction box h1. Similarly, since the probability of the object's existence (0.23) in the output parameters [0.23: 0.02, 0.01, 0.94] for prediction box h2 is less than the probability threshold of 0.5, the business server can also directly filter out the predicted category probabilities of 0.02, 0.01, and 0.94 in the output parameters [0.23: 0.02, 0.01, 0.94], and no longer use these probabilities to determine the object category of the object in prediction box h2.

[0117] like Figure 2aAs shown, the prediction box corresponding to the location feature point d2 includes prediction box e1 and prediction box e2. The output parameters corresponding to the classification feature point m2 are the output parameters corresponding to prediction box e1 and prediction box e2 (that is, the object existence probability and category prediction probability corresponding to prediction box e1 and prediction box e2, respectively). For example, the output parameters corresponding to the classification feature point m2 are [0.98: 0.01, 0.97, 0.03] and [0.99: 0.02, 0.76, 0.56]. Among them, the output parameter [0.98: 0.01, 0.97, 0.03] corresponds to the prediction box e1. The value 0.98 in the output parameter [0.98: 0.01, 0.97, 0.03] is the probability of the object existing in the prediction box e1. The values ​​0.01, 0.97, and 0.03 in the output parameter [0.98: 0.01, 0.97, 0.03] are all predicted category probabilities. Among them, the predicted category corresponding to the value 0.01 is comics, the predicted category corresponding to the value 0.97 is illegal watermark, and the predicted category corresponding to the value 0.03 is letters. Because the probability of the object's existence (0.98) is greater than the probability threshold of 0.5, the business server can determine that an object exists in prediction box e1. Since the maximum predicted category among the predicted category probabilities of 0.01, 0.97, and 0.03 is 0.97, the business server can determine the predicted category "violation watermark" corresponding to the maximum predicted category of 0.97 as the object category of the object in prediction box e1. Similarly, in the output parameters [0.99: 0.02, 0.76, 0.56], because the probability of the object's existence (0.99) is greater than the probability threshold of 0.5, the business server can determine that an object exists in prediction box e2. Therefore, the business server can determine the maximum predicted category probability of 0.76 among the predicted category probabilities of 0.02, 0.76, and 0.56, and determine the predicted category "violation watermark" corresponding to the maximum predicted category probability as the object category of the object in prediction box e2.

[0118] Similarly, the business server can determine, based on the output parameters of the classification feature point m3, that the prediction box r1 corresponding to the location feature point d3 contains the object "monkey", and the prediction box r2 contains the object "monkey"; similarly, the business server can determine, based on the output parameters of the classification feature point m4, that the prediction box f1 corresponding to the location feature point d4 does not contain an object, and the prediction box f2 does not contain an object.

[0119] Furthermore, based on the output parameters [position offset, prediction box width, prediction box height] of position feature points d1, d2, d3, and d4, the position information of prediction boxes h1, h2, e1, e2, r1, r2, f1, and f2 in the target image 20c is determined. The position information of these prediction boxes in the target image 20c can be as follows: Figure 2a As shown.

[0120] It can be seen that the objects in both prediction boxes e1 and e2 are the illegal watermark "www.......com". Since prediction box e1 includes the entire content of the illegal watermark "www.......com", while prediction box e2 does not include the entire content of the illegal watermark "www.......com", the score of prediction box e1 (which can be understood as the coverage of the object) is higher than that of prediction box e2. Furthermore, since the overlap rate between prediction boxes e1 and e2 is very high, the business server can delete prediction box e2 and determine prediction box e1 as the location information of the illegal watermark "www.......com" in grid pixel block g2 (i.e., target image 20c).

[0121] Similarly, the business server can determine, based on the output parameters of the classification feature point m3, that the predicted bounding box r1 corresponding to the location feature point d3 contains the object "monkey," and the predicted bounding box r1 represents the location information of the "monkey" object in the grid pixel block g3 (i.e., in the target image 20c). Furthermore, the business server can determine that image 20a contains the sensitive watermark "www…….com" and determine the location information of the sensitive watermark "www…….com" in image 20a, as well as the location information of the "monkey" in image 20a. Simultaneously, the business server can delete predicted bounding boxes that do not contain the object (e.g., predicted bounding boxes h1, h2, f1, and f2), thus obtaining the recognition result for the target image 20c (predicted bounding box r1 contains the object "monkey," and predicted bounding box e1 contains the sensitive watermark "www…….com"). The business server can return the recognition result to the backend server of user terminal A. User A can view the result on the display page of user terminal A and perform subsequent processing, such as deleting the illegal watermark "www……com" to ensure that image 20a is a legal image.

[0122] For ease of understanding, please refer to the following: Figure 2d , Figure 2d This is an application scenario diagram provided in an embodiment of this application. Wherein, as... Figure 2d The service server shown can be the aforementioned service server 1000, and as follows: Figure 2dThe terminal A shown can be the one described above. Figure 1 Any user terminal selected in the user terminal cluster of the corresponding embodiment, for example, the user terminal can be the aforementioned user terminal 100a.

[0123] like Figure 2d As shown, user A uploaded image 20a (as described above) through user terminal A. Figure 2a In the corresponding embodiment, image 20a), user terminal A can send image 20a to the service server. After receiving image 20a, the service server can identify image 20a and determine the target object contained in image 20a and the location information of the target object in image 20a. Figure 2d As shown, the business server can determine that image 20a contains the illegal watermark "www...com" and the target object "monkey," and the business server outlines the location of the illegal watermark "www...com" and the target object "monkey" in image 20a. Subsequently, the business server can return this recognition result to the backend server corresponding to user terminal A. User terminal A can create an object selection interface based on the recognition result returned by the business server, allowing user A to select the object they want to delete. Figure 2d As shown, in the object selection interface, user terminal A can mark the illegal watermark "www...com" (e.g., gray out the area S where the illegal watermark is located) so that user A can know that the object in area S is an illegal object. Figure 2d As shown, in the object selection interface, user A selects the illegal watermark "www...com" in region S. User terminal A can then delete the illegal watermark "www...com" from image 20a. After successful deletion, user A can view the "Deleted successfully" message on the deletion success interface. If user A has no other objects to delete, user A can click "OK". This click triggers an operation that redirects user terminal A to the image viewing interface, where user A can view the image after the object deletion, as shown below. Figure 2d As shown, image 20a in the image viewing interface does not contain any object violating the rules. Image 20a after removing the violating watermark "www……com" is a valid image.

[0124] For easier understanding, please refer to Figure 2e , Figure 2e This is an application scenario diagram provided in an embodiment of this application. For example... Figure 2e The service server shown can be the aforementioned service server 1000, and as follows: Figure 2e The terminal B shown can be the one described above. Figure 1Any user terminal selected in the user terminal cluster of the corresponding embodiment, for example, the user terminal can be the aforementioned user terminal 100b.

[0125] like Figure 2e As shown, User B can be an image reviewer, capable of reviewing images uploaded by other users. Figure 2e As shown, user B can send images 20a uploaded by other users (as described above) through the user terminal. Figure 2a The image 20a) in the corresponding embodiment is sent to the service server to detect whether the image 20a contains a violation object. Figure 2d As shown, after receiving image 20a, the business server can identify image 20a and determine that it contains the illegal watermark "www...com". Therefore, the business server can determine that image 20a is an illegal image. Subsequently, the business server can return the identification result (image 20a is an illegal image, containing the illegal watermark "www...com") to the backend server of user terminal B. After receiving the identification result, user terminal B can query the information of the image provider of the illegal image 20a (e.g., user A) (e.g., name, user photo, home address, etc.) and create a user viewing interface to display user A's information. User B can then view the information of user A who uploaded the illegal image 20a in the user viewing interface and perform subsequent processing (e.g., notifying user A that the uploaded image 20a is an illegal image and has not passed the review).

[0126] For easier understanding, please refer to Figure 3 , Figure 3 This is a schematic flowchart of an image recognition method provided in an embodiment of this application. Figure 3 As shown, the process may include:

[0127] Step S101: Obtain the target image, input the target image into the image recognition model, and extract the image features of the target image in the image recognition model.

[0128] In this application, the target image includes K grid pixels after image partitioning, where K is an integer greater than 1. This can be understood as follows: after obtaining an original image, the original image can be scaled according to an image scaling ratio to obtain a transition image. Then, the transition image is partitioned to obtain the target image including K grid pixels. Figure 2aTaking the corresponding embodiment as an example, the image 20a uploaded by user A is the original image. The size of the original image 20a is 16*8. The obtained image scaling ratio is 8*8. According to the image scaling ratio, the length 16 of the original image 20a can be scaled to a length of 8. According to the ratio of length 16 to length 8, the width 8 of the original image 20a can be scaled to a width of 4. After scaling the length and width, the resulting image size is 8*4. Since it has not yet reached the image scaling ratio of 8*8, the difference can be filled to obtain the transition image 20b. Subsequently, the transition image 20b can be divided into images to obtain the target image 20c, which includes 8*8 = 64 grid pixels.

[0129] Furthermore, the target image can be input into an image recognition model, where its image features can be extracted. For clarity, please refer to [link to related documentation]. Figure 4 , Figure 4 This is a schematic diagram illustrating an image feature extraction method provided in an embodiment of this application. For example... Figure 4As shown, the image recognition model includes a residual network (including a first feature extraction unit, a second feature extraction unit, and a third feature extraction unit), a first convolutional layer, a second convolutional layer, and a third convolutional layer. A target image with 416*416 grid pixels can be input into the image recognition model. Through the first feature extraction unit (including the DBL module, res1 module, res2 module, and res8 module), the pixel features of each grid pixel in the target image can be extracted, resulting in 416*416 first pixel features. The DBL module, short for Darknetconv2d_BN_Leaky, consists of a convolutional layer (Darknetconv2d layer) + a batch normalization (BN) layer + an activation function (Leaky ReLU) layer. The DBL module is a fundamental module in the image recognition model and can be used to perform convolution processing on input data (e.g., the target image). The `resn` module (where n can be 1, 2, 4, or 8) consists of a zero-padding layer, a DBL module, and n `res` units. `resn` is a module in the image recognition model that allows the model to borrow the residual structure of ResNet (Residual Network). This residual structure enables the image recognition model to extract deeper image features from the input data (e.g., the target image). Each `res` unit consists of two DBL modules and an add layer. Both the DBL and `resn` modules can be used to perform convolution processing on the input data. Therefore, the first feature extraction unit, including both the DBL and `resn` modules, can perform convolution on the target image to extract the first pixel features. The image recognition model here can be a YOLO (You Only Look Once) model.

[0130] Subsequently, the 416*416 first pixel features can be input into the second feature extraction unit (including res8), where each first pixel feature can be convolved to obtain 416*416 second pixel features. Then, these 416*416 second pixel features can be input into the third feature extraction unit, where each second pixel feature can be convolved to obtain 416*416 third pixel features. Next, these K third pixel features can be input into the first convolutional layer (including DBL), where each third pixel feature can be convolved to obtain 416*416 initial convolutional pixel features. Based on the first downsampling factor corresponding to the third feature extraction unit (e.g., ...), ... Figure 4(1 / 32 of the original 416*416 initial convolutional pixel features) can be downsampled, that is, the 416*416 initial convolutional pixel features are divided into multiple convolutional pixel feature points. One convolutional pixel feature point is obtained by convolving 32*32 initial convolutional pixel features. Therefore, by dividing, the 416*416 initial convolutional pixel features can be divided into 13*13 convolutional pixel feature points, thus obtaining convolutional pixel features including 13*13 convolutional pixel feature points, such as... Figure 4 As shown, the convolutional pixel feature, which includes 13*13 convolutional pixel feature points, can be determined as the image feature of the target image.

[0131] Optionally, and understandably, the features obtained by downsampling the initial 416*416 convolutional pixel features according to the first downsampling factor (1 / 32) corresponding to the third feature extraction unit can be determined as the first downsampled pixel features (including 13*13 convolutional pixel feature points); subsequently, the features can be determined according to the second downsampling factor (e.g., the second feature extraction unit). Figure 4 As shown in 1 / 16), the aforementioned 416*416 second pixel features are downsampled, that is, the 416*416 second pixel features are divided into multiple pixel feature points. Among them, one convolutional pixel feature point is obtained by convolving 16*16 second pixel features. Therefore, through this division, the 416*416 second pixel features can be divided into 26*26 pixel feature points, thus obtaining a second downsampled pixel feature including 26*26 pixel feature points. Subsequently, the first downsampling factor (1 / 32) and the second downsampling factor (1 / 16) can be used to determine the first downsampling factor (1 / 32). An image is upsampled by a factor of 2. Based on this first image upsampling factor of 2, the aforementioned first downsampled pixel features, including 13*13 convolutional pixel feature points, can be upsampled. This allows the size of the first downsampled pixel features, including 13*13 convolutional pixel feature points, to be magnified by a factor of 2, resulting in upsampled pixel features including 26*26 convolutional pixel sub-feature points. Subsequently, the second downsampled pixel features, including 26*26 pixel feature points, can be concatenated with the upsampled pixel features, including 26*26 convolutional pixel sub-feature points, to obtain the first concatenated feature, such as... Figure 4 As shown, the first stitched feature can be input into the second convolutional layer (including the DBL module). After the first stitched feature is convolved in the second convolutional layer, a convolutional pixel feature including 26*26 convolutional pixel feature points can be obtained. The convolutional pixel feature including 26*26 convolutional pixel feature points can be determined as the image feature of the target image.

[0132] Optionally, it is understood that the features obtained after performing convolution processing on the first concatenated feature in the second convolutional layer can be defined as convolutional concatenated features (including 26*26 convolutional pixel feature points). Subsequently, the third downsampling factor (e.g., based on the first feature extraction unit) can be used to extract the concatenated features. Figure 4 As shown in 1 / 8), the 416*416 first pixel features are downsampled, that is, the 416*416 first pixel features are divided into multiple pixel feature points. Among them, each convolutional pixel feature point consists of 8*8 second pixel features. Therefore, through this division, the 416*416 first pixel features can be divided into 52*52 pixel feature points, thus obtaining a third downsampled pixel feature including 52*52 pixel feature points. Subsequently, the second image can be determined based on the third downsampling factor (1 / 8) and the second downsampling factor (1 / 16). With an upsampling factor of 2, the convolutional stitched feature comprising 26*26 convolutional pixel feature points can be upsampled according to this second image upsampling factor of 2. This can double the size of the convolutional stitched feature comprising 26*26 convolutional pixel feature points, resulting in an upsampled stitched feature comprising 52*52 convolutional pixel sub-feature points. Subsequently, the third downsampled pixel feature comprising 52*52 pixel feature points can be stitched together with the upsampled stitched feature comprising 52*52 convolutional pixel sub-feature points to obtain the second stitched feature, as shown below. Figure 4 As shown, the second stitched feature can be input into the third convolutional layer (including the DBL module). After the second stitched feature is convolved in the third convolutional layer, a convolutional pixel feature including 52*52 convolutional pixel feature points can be obtained. The convolutional pixel feature including 52*52 convolutional pixel feature points can be determined as the image feature of the target image.

[0133] In summary, the image features of the target image can be features including 13*13 convolutional pixel feature points, features including 26*26 convolutional pixel feature points, or features including 52*52 convolutional pixel feature points. Among these, the image features with 26*26 convolutional pixel feature points are more specific than those with 13*13 convolutional pixel feature points because they are obtained by upsampling the 13*13 convolutional pixel feature points by a factor of 2. Similarly, the image features with 52*52 convolutional pixel feature points are more specific than those with 26*26 convolutional pixel feature points.

[0134] Step S102: Input the image features into the image classification component and the location prediction component in the image recognition model, respectively; the image classification component contains a first convolutional parameter that focuses on object classification, and the location prediction component contains a second convolutional parameter that focuses on location information prediction.

[0135] In this application, both the image classification component and the location prediction component are included in the image recognition model. The image classification component includes a first convolutional parameter focused on object classification, meaning it can be used to identify the object category of a target object in the target image. The location prediction component includes a second convolutional parameter focused on location information prediction, meaning it can be used to determine the location information of the target object in the target image.

[0136] Step S103: In the image classification component, the image features are convolved using the first convolution parameter to obtain a classification convolution feature map.

[0137] In this application, the first convolution parameter is a parameter in the image classification component, which can be used to perform convolution processing on image features. By using the first convolution parameter, each convolution pixel feature point in the image features can be convolved, resulting in a classification convolution feature containing multiple classification feature points. Each classification feature point is obtained by convolving one convolution pixel feature point. For example, if the image features include 13*13 convolution pixel feature points, then after convolving the image features using the first convolution parameter, a classification convolution feature map containing 13*13 classification feature points can be obtained.

[0138] Step S104: In the location prediction component, the image features are convolved using the second convolution parameter to obtain a location convolution feature map.

[0139] In this application, the second convolution parameter is a parameter in the location prediction component, which can be used to perform convolution processing on image features. The second convolution parameter can be used to perform convolution processing on each convolution pixel feature point in the image features, thereby obtaining a classification convolution feature containing multiple location feature points. Each location feature point is obtained by convolving one convolution pixel feature point. For example, if the image features include 13*13 convolution pixel feature points, then after performing convolution processing on the image features using the second convolution parameter, a location convolution feature map including 13*13 location feature points can be obtained.

[0140] For ease of understanding, please refer to the following: Figure 5 , Figure 5 This is a schematic diagram illustrating how to determine a classification convolutional feature map and a positional convolutional feature map, as provided in an embodiment of this application. Figure 5 As shown, a target image comprising 416*416 grid pixels is input into an image recognition model, which can then extract the image features of the target image. For a detailed explanation of how the image recognition model extracts the image features of the 416*416 grid pixel target image, please refer to the aforementioned... Figure 4 The description of image feature extraction by the image recognition model in the corresponding embodiment is as follows: The image features are input into the image classification component of the image recognition model. The image features are then convolved using the first convolution parameter of the image classification component to obtain a classification convolutional feature map. For example, taking an image feature consisting of 13*13 convolutional pixel feature points as an example... Figure 5 As shown, the image features comprising 13*13 convolutional pixel feature points are input into an image classification component (including a DBL module and a conv module). This image classification component performs convolution processing on each of the 13*13 convolutional pixel feature points, thereby obtaining a classification convolutional feature map comprising 13*13 classification feature points. This image feature is then input into the location prediction component of the image recognition model. The second convolution parameter of the location prediction component performs convolution processing on the image features, resulting in a location convolutional feature map.

[0141] For example, taking an image feature consisting of 13*13 convolutional pixel feature points as an example, such as Figure 5 As shown, the image features comprising 13*13 convolutional pixel feature points are input into a location prediction component (including a DBL module and a conv module). This location prediction component performs convolution processing on each of the 13*13 convolutional pixel feature points, thereby obtaining a location convolutional feature map comprising 13*13 location feature points. Figure 5 As shown, the image features corresponding to a 52*52 classification convolutional feature map (including 52*52 convolutional pixel feature points) are more specific than those corresponding to a 26*26 classification convolutional feature map (including 26*26 convolutional pixel feature points); the image features corresponding to a 26*26 classification convolutional feature map (including 26*26 convolutional pixel feature points) are more specific than those corresponding to a 13*13 classification convolutional feature map (including 13*13 convolutional pixel feature points). Therefore, a 13*13 classification convolutional feature map can be used to classify large objects (e.g., objects occupying a large area) in a target image, a 26*26 classification convolutional feature map can be used to classify medium-sized objects (e.g., objects occupying a suitable area) in a target image, and a 52*52 classification convolutional feature map can be used to classify small objects (e.g., objects occupying a small area) in a target image. Similarly, a 13*13 positional convolutional feature map can be used to determine the positional information of large objects in a target image, a 26*26 classification convolutional feature map can be used to determine the positional information of medium-sized objects in a target image, and a 52*52 classification convolutional feature map can be used to determine the positional information of small objects in a target image.

[0142] Step S105: Based on the classification convolution feature map, output the object category of the target object in the target image; based on the position convolution feature map, output the position information of the target object in the target image.

[0143] In this application, each classification feature point in the classification convolution feature map corresponds to a grid pixel block in the target image, for example, using Figure 2a Taking the corresponding embodiment as an example, Figure 2a The classification feature point m1 in the image corresponds to a 4*4 grid pixel block g1 in the target image 20c. This classification feature point m1 is obtained by performing multiple convolutions on the 4*4 grid pixel block g1 using an image recognition model. During image classification, the probability of object presence output by the classification feature point can be obtained. This probability refers to the probability that a target object (e.g., a person, animal, website address, letter, text, etc.) exists within the prediction box to which the classification feature point belongs. This prediction box is used to predict the position of the target object in the target image. For example, using... Figure 2a Taking the corresponding embodiment as an example, the classification feature point m1 corresponds to the grid pixel block g1, and the 4*4 grid pixel block g1 is included in the 64 grid pixels of the target image 20c. The size of the grid pixel block is determined by the downsampling factor corresponding to the feature extraction unit (including the first feature extraction unit, the second feature extraction unit, and the third feature extraction unit), for example, using the above... Figure 4 Taking the corresponding embodiment as an example, when the third feature extraction unit outputs the third pixel feature, if the third pixel feature is determined as the target pixel feature, the size of the grid pixel block (i.e., 32*32) can be determined according to the downsampling factor (1 / 32) of the third feature extraction unit. That is, the target image including 416*416 grid pixels is divided into a target image including 13*13 grid pixel blocks (each grid pixel block is composed of 32*32 grid pixels).

[0144] Understandably, classification feature points can be obtained by performing multiple convolutions on grid pixel blocks. Each classification feature point can output one or more output parameters, each including the object presence probability and the class prediction probability. Each output parameter corresponds to a prediction bounding box. Furthermore, the class prediction probabilities output by the classification feature point can be obtained. When the object presence probability output by the classification feature point is greater than a probability threshold, it can be determined that an object exists in the prediction bounding box. Then, the highest class prediction probability can be determined from these class prediction probabilities, and the class corresponding to the highest class prediction probability is determined as the object class predicted by the classification feature point. Subsequently, the object class predicted by the classification feature point can be determined as the object class of the prediction bounding box to which the classification feature point belongs (i.e., the object class of the object contained in the prediction bounding box).

[0145] In this application, each location feature point in the location convolution feature map corresponds to a grid pixel block in the target image, for example, using Figure 2a Taking the corresponding embodiment as an example, Figure 2a The convolutional feature point d1 in the image corresponds to the 4*4 grid pixel block g1 in the target image 20c. This positional feature point d1 is obtained by repeatedly convolving the 4*4 grid pixel block g1 by the image recognition model. In other words, by using different convolutional parameters (e.g., the first convolutional parameter and the second convolutional parameter) to convolve the same grid pixel block, the image recognition model can obtain classification feature points and positional feature points respectively. The output parameters of the classification feature points correspond to the prediction boxes corresponding to the positional feature points. Through the output parameters of the classification feature points, it can be determined whether an object exists in the prediction box corresponding to the positional feature point and what object category it belongs to. The output parameters of the positional feature points can determine the position information of the prediction box in the target image (i.e., the position information of the target object in the prediction box). Specifically, the predicted position parameters (including position offset, prediction box width, and prediction box height) corresponding to the positional feature point in the positional convolutional feature map can be obtained. Subsequently, the center position coordinates corresponding to the positional convolutional feature point can be obtained. Based on the predicted position parameters and the center position coordinates, the position information of the target object can be determined. The specific method involves determining the center position information of the predicted bounding box corresponding to the feature point based on the position offset and the center position coordinates. Then, based on the center position information, the width, and the height of the predicted bounding box, the position information of the predicted bounding box in the target image can be determined, and this position information can be used to identify the position information of the target object. The position offset refers to the offset of the vertex position of the predicted bounding box (e.g., the coordinates of the top left and top right corners) relative to the grid pixel block. By adding or subtracting this position offset from the center position coordinates of the grid pixel block, the center position information of the predicted bounding box within the grid pixel block can be obtained.

[0146] For example, as mentioned above Figure 2a Taking the corresponding embodiment as an example, the predicted position parameters output by the position feature point d3 in the positional convolution feature map are [a, w, h], where the predicted bounding box corresponding to the predicted position parameters [a, w, h] is the predicted bounding box r1, and the parameter a is the position offset. Please refer to [link to relevant documentation]. Figure 2cSince the grid pixel block corresponding to feature point d3 is grid pixel block g3, the parameter 'a' can be used to characterize the top-left corner coordinates of the predicted bounding box r1 relative to the center position of grid pixel block g3. Adding parameter 'a' to the center position coordinates of grid pixel block g3 yields the center position information of the predicted bounding box r1 within grid pixel block g3. Based on this center position information, the parameter w (width of the predicted bounding box r1), and the parameter h (height of the predicted bounding box r1), the position information of the predicted bounding box r1 within grid pixel block g3 can be determined. Furthermore, since the predicted bounding box r1 has the highest score (highest coverage of the object) among the predicted bounding boxes corresponding to grid pixel block g3, the position information of the predicted bounding box r1 can be determined as the position information of the target object within grid pixel block g3.

[0147] In this embodiment, after extracting the image features of the target image, the image features are input into the image classification component and the location prediction component respectively. The image features are convolved by the first convolution parameter in the image classification component to obtain a classification convolution feature map. The object category of the target object in the target image can then be output based on the classification convolution feature map. At the same time, the image features are convolved by the second convolution parameter in the location prediction component to obtain a location convolution feature map. The location information of the target object in the target image can then be output based on the location convolution feature map. Because the first convolution parameter in the image classification component focuses on object classification, while the second convolution parameter in the location prediction component focuses on location information prediction, and these two parameters are independent of each other and have different focuses, using the first convolution parameter to classify image features can improve classification accuracy, and using the second convolution parameter to predict location information can improve location information prediction accuracy. Furthermore, since image features are shared by both the image classification and location prediction components, there is no need to perform two calculations on image features. This not only improves classification accuracy but also saves computational resources, thereby improving recognition efficiency.

[0148] Further, please see Figure 6 , Figure 6 This is a schematic diagram of a model training process provided in an embodiment of this application. For example... Figure 6 As shown, the process may include:

[0149] Step S201: Obtain image samples, input the image samples into the sample image recognition model, and extract the sample features of the image samples in the sample image recognition model.

[0150] In this application, to improve the accuracy of the image recognition model in predicting the object category and location information of the target object, the image recognition model can be trained and adjusted to achieve its optimal state. The image recognition model here can include the YOLO (You Only Look Once) model, such as YOLO_v1, YOLO_v2, and YOLO_v3. Specifically, YOLO_v1, YOLO_v2, and YOLO_v3 models can all extract image features from the input image using a feature extraction network (e.g., residual network + convolutional layers), and then use these image features as shared features, respectively inputting them into the image classification component and the location prediction component of the image recognition model.

[0151] After obtaining image samples, they can be input into a sample image recognition model. Through the residual network and convolutional layers in this model, deep image features (sample features) can be effectively extracted from the image samples. For the specific implementation of the sample image recognition model in extracting sample features, please refer to the above. Figure 4 The description of the image features extracted from the target image by the image recognition model in the corresponding embodiments will not be repeated here.

[0152] Step S202: Input the sample features into the sample image classification component and the sample location prediction component of the sample image recognition model, respectively; the sample image classification component contains a first sample convolution parameter that focuses on object classification, and the sample location prediction component contains a second sample convolution parameter that focuses on location information prediction.

[0153] Step S203: In the sample image classification component, the sample features are convolved using the first sample convolution parameters to obtain the sample classification convolution feature map.

[0154] Step S204: In the sample location prediction component, the sample features are convolved using the second sample convolution parameter to obtain the sample location convolution feature map.

[0155] Step S205: Based on the sample classification convolutional feature map, determine the predicted object category of the sample object in the image sample, and based on the sample position convolutional feature map, determine the predicted position information of the sample object.

[0156] For the specific implementation of steps S202-S205 in this application, please refer to the above. Figure 3 The descriptions of steps S102-S105 in the corresponding embodiments will not be repeated here.

[0157] Step S206: Obtain the object category label of the sample object, obtain the location label information of the sample object, and determine the total loss value based on the predicted object category, object category label, predicted location information, and location label information.

[0158] In this application, a classification loss value can be determined based on the object category label of the sample object and the predicted object category; a location loss value can be determined based on the location label information (true location information) of the sample object and the predicted location information; and a total loss value for image classification and location prediction can be generated based on the classification loss value and the location loss value. For example, the classification loss value and the location loss value can be added together, and the result of the addition can be determined as the total loss value.

[0159] Step S207: Adjust the first sample convolution parameters and the second sample convolution parameters according to the total loss value to obtain the first convolution parameters corresponding to the first sample convolution parameters and the second convolution parameters corresponding to the second sample convolution parameters.

[0160] In this application, the first sample convolution parameter in the sample image classification component can be adjusted based on the total loss value. Specifically, the partial derivative between the total loss value and the predicted object category can be calculated, and the first sample convolution parameter can be adjusted based on the partial derivative to obtain the first convolution parameter. Alternatively, the second sample convolution parameter in the sample location prediction component can be adjusted based on the total loss value. Specifically, the partial derivative between the total loss value and the predicted location information can be calculated, and the second sample convolution parameter can be adjusted based on the partial derivative to obtain the second convolution parameter.

[0161] Step S208: The sample image classification component including the first convolution parameter is determined as the image classification component; the sample position prediction component including the second convolution parameter is determined as the position prediction component; and the sample image recognition model including the image classification component and the position prediction component is determined as the image recognition model.

[0162] In this application, the parameters in the residual network and convolutional layers of the image recognition model can be adjusted using the total loss value. This allows the image features extracted by the residual network and convolutional layers to more specifically represent the target image. After adjustment, the image recognition model can be used in applications. In this model, the image classification component containing the first convolutional parameter can more accurately identify the object category of the target object in the target image, and the location prediction component containing the second convolutional parameter can more accurately determine the location information of the target object.

[0163] In this embodiment, after extracting the image features of the target image, the image features are input into the image classification component and the location prediction component respectively. The image features are convolved by the first convolution parameter in the image classification component to obtain a classification convolution feature map. The object category of the target object in the target image can then be output based on the classification convolution feature map. At the same time, the image features are convolved by the second convolution parameter in the location prediction component to obtain a location convolution feature map. The location information of the target object in the target image can then be output based on the location convolution feature map. Because the first convolution parameter in the image classification component focuses on object classification, while the second convolution parameter in the location prediction component focuses on location information prediction, and these two parameters are independent of each other and have different focuses, using the first convolution parameter to classify image features can improve classification accuracy, and using the second convolution parameter to predict location information can improve location information prediction accuracy. Furthermore, since image features are shared by both the image classification and location prediction components, there is no need to perform two calculations on image features. This not only improves classification accuracy but also saves computational resources, thereby improving recognition efficiency.

[0164] Further, please see Figure 7 , Figure 7 This is a schematic diagram of the structure of an image recognition device provided in an embodiment of this application. The image recognition device described above can be a computer program (including program code) running on a computer device, for example, the image recognition device is an application software; the device can be used to execute the corresponding steps in the method provided in the embodiment of this application. The image recognition device 1 may include: an image acquisition module 11, a feature extraction module 12, a feature input module 13, a first feature convolution module 14, a second feature convolution module 15, a category output module 16, and a position output module 17.

[0165] Image acquisition module 11 is used to acquire the target image and input the target image into the image recognition model;

[0166] Feature extraction module 12 is used to extract image features of the target image in the image recognition model;

[0167] The feature input module 13 is used to input image features into the image classification component and the location prediction component in the image recognition model, respectively; the image classification component contains a first convolution parameter that focuses on object classification, and the location prediction component contains a second convolution parameter that focuses on location information prediction.

[0168] The first feature convolution module 14 is used in the image classification component to perform convolution processing on image features through the first convolution parameters to obtain a classification convolution feature map.

[0169] The second feature convolution module 15 is used to perform convolution processing on image features through the second convolution parameters in the position prediction component to obtain a position convolution feature map.

[0170] The category output module 16 is used to output the object category of the target object in the target image based on the classification convolution feature map;

[0171] The position output module 17 is used to output the position information of the target object in the target image based on the position convolution feature map.

[0172] The specific implementation methods of the image acquisition module 11, feature extraction module 12, feature input module 13, first feature convolution module 14, second feature convolution module 15, category output module 16, and position output module 17 can be found above. Figure 3 The descriptions of steps S101-S105 in the corresponding embodiments will not be repeated here.

[0173] The target image consists of K grid pixels after image division. The image recognition model includes a residual network, which comprises a first feature extraction unit, a second feature extraction unit, a third feature extraction unit, and a convolutional layer. K is an integer greater than 1.

[0174] Please see Figure 7 The feature extraction module 12 may include: a first pixel feature output unit 121, a second pixel feature output unit 122, a third pixel feature output unit 123, a target pixel feature selection unit 124, a feature convolution unit 125, and an image feature generation unit 126.

[0175] The first pixel feature output unit 121 is used to input the target image into the first feature extraction unit, and extract the pixel features of each grid pixel in the target image in the first feature extraction unit to obtain K first pixel features;

[0176] The second pixel feature output unit 122 is used to input K first pixel features into the second feature extraction unit, and to perform convolution processing on each first pixel feature in the second feature extraction unit to obtain K second pixel features.

[0177] The third pixel feature output unit 123 is used to input K second pixel features into the third feature extraction unit, and to perform convolution processing on each second pixel feature in the third feature extraction unit to obtain K third pixel features.

[0178] The target pixel feature selection unit 124 is used to select K target pixel features from K first pixel features, K second pixel features and K third pixel features;

[0179] The feature convolution unit 125 is used to input K target pixel features into the convolutional layer, and perform convolution processing on the K target pixel features in the convolutional layer to obtain convolutional pixel features;

[0180] The image feature generation unit 126 is used to generate image features of the target image based on the convolution pixel features.

[0181] The specific implementation methods of the first pixel feature output unit 121, the second pixel feature output unit 122, the third pixel feature output unit 123, the target pixel feature selection unit 124, the feature convolution unit 125, and the image feature generation unit 126 can be found in the above description. Figure 3 The description of step S101 in the corresponding embodiment will not be repeated here.

[0182] Please see Figure 7 The image recognition device 1 may include: an image acquisition module 11, a feature extraction module 12, a feature input module 13, a first feature convolution module 14, a second feature convolution module 15, a category output module 16, and a position output module 17. It may also include: a scale acquisition module 18, an image scaling module 19, and an image segmentation module 20.

[0183] The scaling module 18 is used to acquire the original image and obtain the image scaling ratio.

[0184] Image scaling module 19 is used to scale the original image according to the image scaling ratio to obtain a transition image;

[0185] The image segmentation module 20 is used to segment the transition image to obtain a target image including K grid pixels.

[0186] The specific implementation methods of the ratio acquisition module 18, the image scaling module 19, and the image segmentation module 20 can be found in the foregoing. Figure 3 The description of acquiring the target image in step S101 of the corresponding embodiment will not be repeated here.

[0187] Among them, the K target pixel features are K third pixel features; the convolutional layer includes the first convolutional layer;

[0188] Please see Figure 7 The feature convolution unit 125 may include: a first convolution subunit 1251 and a first downsampling subunit 1252.

[0189] The first convolutional subunit 1251 is used to input K third pixel features into the first convolutional layer, and to perform convolution processing on each third pixel feature in the first convolutional layer to obtain K initial convolutional pixel features.

[0190] The first downsampling subunit 1252 is used to obtain the first downsampling factor corresponding to the third feature extraction unit, and to downsample the K initial convolutional pixel features according to the first downsampling factor to obtain the convolutional pixel features.

[0191] The specific implementation of the first convolution sub-unit 1251 and the first downsampling sub-unit 1252 can be found in the foregoing. Figure 3 The description of step S101 in the corresponding embodiment will not be repeated here.

[0192] The convolutional layer includes a first convolutional layer and a second convolutional layer;

[0193] Please see Figure 7 The feature convolution unit 125 may include: a second convolution subunit 1253, a second downsampling subunit 1254, a third downsampling subunit 1255, an upsampling subunit 1256, and a feature splicing subunit 1257.

[0194] The second convolutional subunit 1253 is used to input K third pixel features into the first convolutional layer, and to perform convolution processing on each third pixel feature in the first convolutional layer to obtain K initial convolutional pixel features.

[0195] The second downsampling subunit 1254 is used to obtain the first downsampling factor corresponding to the third feature extraction unit, and downsample the K initial convolutional pixel features according to the first downsampling factor to obtain the first downsampled pixel features.

[0196] The third downsampling subunit 1255 is used to obtain the second downsampling factor corresponding to the second feature extraction unit, and downsample the K second pixel features according to the second downsampling factor to obtain the second downsampled pixel features.

[0197] The upsampling subunit 1256 is used to determine the upsampling factor of the first image based on the first downsampling factor and the second downsampling factor, and to upsample the first downsampled pixel features based on the first image upsampling factor to obtain the upsampled pixel features;

[0198] The feature splicing subunit 1257 is used to splice the second downsampled pixel features with the upsampled pixel features if the K target pixel features are K second pixel features, to obtain the first spliced ​​feature, and input the first spliced ​​feature into the second convolutional layer, and perform convolution processing on the first spliced ​​feature in the second convolutional layer to obtain the convolutional pixel feature.

[0199] The specific implementation methods of the second convolution subunit 1253, the second downsampling subunit 1254, the third downsampling subunit 1255, the upsampling subunit 1256, and the feature concatenation subunit 1257 can be found in the aforementioned documents. Figure 3The description of step S101 in the corresponding embodiment will not be repeated here.

[0200] The convolutional layer also includes a third convolutional layer;

[0201] Please see Figure 7 The image recognition device 1 may include: an image acquisition module 11, a feature extraction module 12, a feature input module 13, a first feature convolution module 14, a second feature convolution module 15, a category output module 16, a position output module 17, a ratio acquisition module 18, an image scaling module 19, and an image segmentation module 20. It may also include: a first feature stitching module 21, a feature downsampling module 22, a feature upsampling module 23, and a second feature stitching module 24.

[0202] The first feature concatenation module 21 is used to concatenate the second downsampled pixel features with the upsampled pixel features if the K target pixel features are K first pixel features, to obtain the first concatenated feature, input the first concatenated feature into the second convolutional layer, and perform convolution processing on the first concatenated feature in the second convolutional layer to obtain the convolutional concatenated feature.

[0203] The feature downsampling module 22 is used to obtain the third downsampling factor corresponding to the first feature extraction unit, and downsample the K first pixel features according to the third downsampling factor to obtain the third downsampled pixel features.

[0204] The feature upsampling module 23 is used to determine the upsampling factor of the second image based on the third downsampling factor and the second downsampling factor, and upsample the convolutional concatenation features according to the upsampling factor of the second image to obtain the upsampled concatenation features;

[0205] The second feature concatenation module 24 is used to concatenate the third downsampled pixel features with the upsampled concatenation features to obtain the second concatenation features. The second concatenation features are then input into the third convolutional layer, where they are convolved to obtain the convolutional pixel features.

[0206] The specific implementation methods of the first feature concatenation module 21, the feature downsampling module 22, the feature upsampling module 23, and the second feature concatenation module 24 can be found in the above description. Figure 3 The description of step S101 in the corresponding embodiment will not be repeated here.

[0207] Please see Figure 7 The category output module 16 may include: an existence probability acquisition unit 161, a prediction probability acquisition unit 162, and a category determination unit 163.

[0208] The existence probability acquisition unit 161 is used to acquire classification feature points in the classification convolution feature map and acquire the existence probability of the object corresponding to the classification feature points; the object existence probability refers to the probability that the target object exists in the prediction box to which the classification feature point belongs; the prediction box is used to predict the position of the target object in the target image.

[0209] The prediction probability acquisition unit 162 is used to acquire the category prediction probability corresponding to the classification feature point;

[0210] The category determination unit 163 is used to obtain the maximum category prediction probability from the category prediction probability if the probability of an object is greater than the probability threshold, and determine the category corresponding to the maximum category prediction probability as the object category predicted by the classification feature point.

[0211] The aforementioned category determination unit 163 is also used to determine the object category predicted by the classification feature point as the object category of the prediction box to which the classification feature point belongs.

[0212] The specific implementation methods of the probability acquisition unit 161, the prediction probability acquisition unit 162, and the category determination unit 163 can be found in the above description. Figure 3 The description of step S105 in the corresponding embodiment will not be repeated here.

[0213] Please see Figure 7 The position output module 17 may include: a position parameter acquisition unit 171, a pixel block acquisition unit 172, a center coordinate acquisition unit 173, and a position information determination unit 174.

[0214] The position parameter acquisition unit 171 is used to acquire position feature points in the position convolution feature map and acquire the predicted position parameters corresponding to the position feature points.

[0215] The pixel block acquisition unit 172 is used to acquire the grid pixel block corresponding to the position feature point in K grid pixels; the position feature point is obtained by convolution of the grid pixel block, and the size of the grid pixel block is determined by the downsampling factor corresponding to the K target pixel features;

[0216] The center coordinate acquisition unit 173 is used to acquire the coordinates of the center position of the grid pixel block;

[0217] The location information determination unit 174 is used to determine the location information of the target object based on the predicted location parameters and the center location coordinates.

[0218] The specific implementation methods of the position parameter acquisition unit 171, pixel block acquisition unit 172, center coordinate acquisition unit 173, and position information determination unit 174 can be found in the above description. Figure 3 The description of step S105 in the corresponding embodiment will not be repeated here.

[0219] The predicted position parameters include the position offset, the width of the predicted bounding box, and the height of the predicted bounding box; the predicted bounding box is used to predict the position of the target object in the target image.

[0220] Please see Figure 7 The location information determination unit 174 may include: a center location determination subunit 1741 and a location information determination subunit 1742.

[0221] The center position determination subunit 1741 is used to determine the center position information corresponding to the prediction box based on the position offset and the center position coordinates.

[0222] The position information determination subunit 1742 is used to determine the position information of the prediction box in the target image based on the center position information, the width of the prediction box and the height of the prediction box, and to determine the position information of the prediction box in the target image as the position information of the target object.

[0223] The specific implementation methods of the center position determination subunit 1741 and the position information determination subunit 1742 can be found in the above description. Figure 3 The description of step S105 in the corresponding embodiment will not be repeated here.

[0224] Please see Figure 7 The image recognition device 1 may include: an image acquisition module 11, a feature extraction module 12, a feature input module 13, a first feature convolution module 14, a second feature convolution module 15, a category output module 16, a position output module 17, a scale acquisition module 18, an image scaling module 19, an image segmentation module 20, a first feature stitching module 21, a feature downsampling module 22, a feature upsampling module 23, and a second feature stitching module 24. It may also include: a sample acquisition module 25, a sample input module 26, a classification convolution module 27, a position convolution module 28, a result determination module 29, a loss value determination module 30, a parameter adjustment module 31, and a model determination module 32.

[0225] The sample acquisition module 25 is used to acquire image samples, input the image samples into the sample image recognition model, and extract the sample features of the image samples in the sample image recognition model;

[0226] The sample input module 26 is used to input sample features into the sample image classification component and the sample location prediction component of the sample image recognition model, respectively; the sample image classification component contains a first sample convolution parameter that focuses on object classification, and the sample location prediction component contains a second sample convolution parameter that focuses on location information prediction.

[0227] The classification convolution module 27 is used to perform convolution processing on the sample features through the first sample convolution parameters in the sample image classification component to obtain the sample classification convolution feature map.

[0228] The location convolution module 28 is used to perform convolution processing on the sample features through the second sample convolution parameters in the sample location prediction component to obtain the sample location convolution feature map.

[0229] The result determination module 29 is used to determine the predicted object category of the sample object in the image sample based on the sample classification convolutional feature map, and to determine the predicted location information of the sample object based on the sample location convolutional feature map.

[0230] The loss value determination module 30 is used to obtain the object category label of the sample object, obtain the location label information of the sample object, and determine the total loss value based on the predicted object category, object category label, predicted location information and location label information;

[0231] The parameter adjustment module 31 is used to adjust the first sample convolution parameter and the second sample convolution parameter according to the total loss value, so as to obtain the first convolution parameter corresponding to the first sample convolution parameter and the second convolution parameter corresponding to the second sample convolution parameter.

[0232] The model determination module 32 is used to determine the sample image classification component including the first convolution parameter as an image classification component, the sample position prediction component including the second convolution parameter as a position prediction component, and the sample image recognition model including the image classification component and the position prediction component as an image recognition model.

[0233] The specific implementation methods of the sample acquisition module 25, sample input module 26, classification convolution module 27, positional convolution module 28, result determination module 29, loss value determination module 30, parameter adjustment module 31, and model determination module 32 can be found above. Figure 6 The descriptions of steps S201-S208 in the corresponding embodiments will not be repeated here.

[0234] Please see Figure 7 The loss value determination module 30 may include: a classification loss determination unit 301, a location loss determination unit 302, and a loss value generation unit 303.

[0235] The classification loss determination unit 301 is used to determine the classification loss value based on the predicted object category and the object category label;

[0236] The location loss determination unit 302 is used to determine the location loss value based on the location label information and the predicted location information;

[0237] The loss value generation unit 303 is used to generate a total loss value based on the classification loss value and the location loss value.

[0238] The specific implementation methods of the classification loss determination unit 301, the location loss determination unit 302, and the loss value generation unit 303 can be found in the above description. Figure 6 The description of step S206 in the corresponding embodiment will not be repeated here.

[0239] Please see Figure 7 The parameter adjustment module 31 may include a first differentiation unit 311 and a second differentiation unit 312.

[0240] The first derivative unit 311 is used to determine the first partial derivative between the total loss value and the predicted object category, and to adjust the first sample convolution parameters according to the first partial derivative to obtain the first convolution parameters.

[0241] The second derivative unit 312 is used to determine the second partial derivative between the total loss value and the predicted location information, and to adjust the second sample convolution parameters according to the second partial derivative to obtain the second convolution parameters.

[0242] The specific implementation methods of the first differentiation unit 311 and the second differentiation unit 312 can be found in the above description. Figure 6 The description of step S207 in the corresponding embodiment will not be repeated here.

[0243] In this embodiment, after extracting the image features of the target image, the image features are input into the image classification component and the location prediction component respectively. The image features are convolved by the first convolution parameter in the image classification component to obtain a classification convolution feature map. The object category of the target object in the target image can then be output based on the classification convolution feature map. At the same time, the image features are convolved by the second convolution parameter in the location prediction component to obtain a location convolution feature map. The location information of the target object in the target image can then be output based on the location convolution feature map. Because the first convolution parameter in the image classification component focuses on object classification, while the second convolution parameter in the location prediction component focuses on location information prediction, and these two parameters are independent of each other and have different focuses, using the first convolution parameter to classify image features can improve classification accuracy, and using the second convolution parameter to predict location information can improve location information prediction accuracy. Furthermore, since image features are shared by both the image classification and location prediction components, there is no need to perform two calculations on image features. This not only improves classification accuracy but also saves computational resources, thereby improving recognition efficiency.

[0244] Further, please see Figure 8 , Figure 8 This is a schematic diagram of a computer device provided in an embodiment of this application. Figure 8 As shown, the aforementioned computer device 1000 may include: at least one processor 1001, such as a CPU; at least one network interface 1004; a user interface 1003; a memory 1005; and at least one communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and a keyboard. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as at least one disk drive. Optionally, the memory 1005 may also be at least one storage device located remotely from the aforementioned processor 1001. Figure 8 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a device control application program.

[0245] exist Figure 8 In the computer device 1000 shown, the network interface 1004 is mainly used for network communication with the business server; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to achieve:

[0246] Acquire the target image, input the target image into the image recognition model, and extract the image features of the target image in the image recognition model;

[0247] Image features are input into the image classification component and the location prediction component in the image recognition model, respectively; the image classification component contains a first convolutional parameter that focuses on object classification, and the location prediction component contains a second convolutional parameter that focuses on location information prediction.

[0248] In the image classification component, the image features are convolved using the first convolution parameter to obtain a classification convolution feature map.

[0249] In the location prediction component, the image features are convolved using the second convolution parameter to obtain the location convolution feature map.

[0250] Based on the classification convolutional feature map, the object category of the target object in the target image is output, and based on the position convolutional feature map, the position information of the target object in the target image is output.

[0251] It should be understood that the computer device 1000 described in the embodiments of this application can execute the foregoing text. Figures 3 to 6 The description of the image recognition method in the corresponding embodiments can also be performed as described above. Figure 7 The description of the image recognition device 1 in the corresponding embodiments will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated.

[0252] Furthermore, it should be noted that this application embodiment also provides a computer-readable storage medium, which stores a computer program executed by the image recognition computer device 1000 mentioned above. The computer program includes program instructions, and when the processor executes the program instructions, it can execute the aforementioned... Figures 3 to 6 The description of the image recognition method in the corresponding embodiments is already provided and will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the computer-readable storage medium embodiments related to this application, please refer to the description of the method embodiments of this application.

[0253] The terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules not listed, or may optionally include other step units inherent to these processes, methods, apparatuses, products, or devices.

[0254] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0255] The methods and related apparatuses provided in this application are described with reference to the method flowcharts and / or structural diagrams provided in this application. Specifically, each block of the method flowchart and / or structural diagram, as well as combinations of blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to create a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the process. Figure 1 A schematic diagram of one or more processes and / or structures. Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 A schematic diagram of one or more processes and / or structures. Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 A process or multiple processes and / or structures illustrate the steps of the functions specified in one or more boxes.

[0256] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. An image recognition method, characterized in that, include: Acquire a target image, input the target image into an image recognition model, and extract the shared image features of the target image in the image recognition model; The shared image features are respectively input into the image classification component and the location prediction component in the image recognition model; the image classification component includes a first convolutional parameter that focuses on object classification, and the location prediction component includes a second convolutional parameter that focuses on location information prediction. The first convolutional parameter and the second convolutional parameter are independent of each other and have different focuses. In the image classification component, the shared image features are convolved using the first convolution parameter to obtain a classification convolution feature map. The classification convolution feature map contains multiple classification feature points. The output parameters corresponding to each classification feature point include the object existence probability and the category prediction probability. The object existence probability is the probability that a target object exists in the prediction box to which the classification feature point belongs. The prediction box is used to predict the position of the target object in the target image. In the location prediction component, the shared image features are convolved using the second convolution parameter to obtain a location convolution feature map. The location convolution feature map contains multiple location feature points, and the output parameters corresponding to each location feature point include predicted location parameters. When the classification feature point and the location feature point correspond to the same network pixel block, one output parameter of the classification feature point corresponds to one output parameter of the location feature point. Based on the object existence probability and category prediction probability corresponding to each classification feature point in the classification convolution feature map, the object category of the prediction box to which each classification feature point belongs is output. Based on the predicted position parameters corresponding to each position feature point in the positional convolutional feature map and the center position coordinates of the network pixel block corresponding to the position feature point, the position information of the target object in the target image is output.

2. The method according to claim 1, characterized in that, The target image comprises K grid pixels after image division, and the image recognition model includes a residual network, which includes a first feature extraction unit, a second feature extraction unit, a third feature extraction unit, and a convolutional layer; K is an integer greater than 1. Extracting shared image features of the target image from the image recognition model includes: The target image is input into the first feature extraction unit, and the pixel features of each grid pixel in the target image are extracted in the first feature extraction unit to obtain K first pixel features; The K first pixel features are input into the second feature extraction unit, and each first pixel feature is convolved in the second feature extraction unit to obtain K second pixel features; The K second pixel features are input into the third feature extraction unit, and each second pixel feature is convolved in the third feature extraction unit to obtain K third pixel features. From the K first pixel features, the K second pixel features, and the K third pixel features, select K target pixel features; The K target pixel features are input into the convolutional layer, and the K target pixel features are convolved in the convolutional layer to obtain convolutional pixel features; Based on the convolutional pixel features, shared image features of the target image are generated.

3. The method according to claim 2, characterized in that, Also includes: Get the original image and get the image scaling ratio; Based on the image scaling ratio, the original image is scaled to obtain a transition image; The transition image is divided into images to obtain a target image comprising K grid pixels.

4. The method according to claim 2, characterized in that, The K target pixel features are the K third pixel features; the convolutional layer includes a first convolutional layer; The step of inputting the K target pixel features into the convolutional layer and performing convolution processing on the K target pixel features in the convolutional layer to obtain convolutional pixel features includes: The K third pixel features are input into the first convolutional layer, and each third pixel feature is convolved in the first convolutional layer to obtain K initial convolutional pixel features. Obtain the first downsampling factor corresponding to the third feature extraction unit, and downsample the K initial convolutional pixel features according to the first downsampling factor to obtain the convolutional pixel features.

5. The method according to claim 2, characterized in that, The convolutional layer includes a first convolutional layer and a second convolutional layer; The step of inputting the K target pixel features into the convolutional layer and performing convolution processing on the K target pixel features in the convolutional layer to obtain convolutional pixel features includes: The K third pixel features are input into the first convolutional layer, and each third pixel feature is convolved in the first convolutional layer to obtain K initial convolutional pixel features. Obtain the first downsampling factor corresponding to the third feature extraction unit, and downsample the K initial convolutional pixel features according to the first downsampling factor to obtain the first downsampled pixel features; Obtain the second downsampling factor corresponding to the second feature extraction unit, and downsample the K second pixel features according to the second downsampling factor to obtain the second downsampled pixel features; Based on the first downsampling factor and the second downsampling factor, the upsampling factor of the first image is determined, and the first downsampled pixel features are upsampled based on the first image upsampling factor to obtain the upsampled pixel features; If the K target pixel features are the K second pixel features, then the second downsampled pixel features are concatenated with the upsampled pixel features to obtain a first concatenated feature. The first concatenated feature is then input into the second convolutional layer, where it is convolved to obtain a convolutional pixel feature.

6. The method according to claim 5, characterized in that, The convolutional layer also includes a third convolutional layer; The method further includes: If the K target pixel features are the K first pixel features, then the second downsampled pixel features are concatenated with the upsampled pixel features to obtain a first concatenated feature. The first concatenated feature is then input into the second convolutional layer, and convolution processing is performed on the first concatenated feature in the second convolutional layer to obtain a convolutional concatenated feature. Obtain the third downsampling factor corresponding to the first feature extraction unit, and downsample the K first pixel features according to the third downsampling factor to obtain the third downsampled pixel features; Based on the third downsampling factor and the second downsampling factor, the upsampling factor of the second image is determined, and the convolutional stitching features are upsampled according to the second image upsampling factor to obtain the upsampled stitching features; The third downsampled pixel feature is concatenated with the upsampled concatenated feature to obtain the second concatenated feature. The second concatenated feature is then input into the third convolutional layer, where it undergoes convolution processing to obtain the convolutional pixel feature.

7. The method according to claim 2, characterized in that, Based on the object existence probability and category prediction probability corresponding to each classification feature point in the classification convolutional feature map, the object category of the prediction box to which each classification feature point belongs is output, including: If the probability of the object is greater than the probability threshold, then the maximum category prediction probability is obtained from the category prediction probabilities, and the category corresponding to the maximum category prediction probability is determined as the object category predicted by the classification feature point. The object category predicted by the classification feature point is determined as the object category of the prediction box to which the classification feature point belongs.

8. The method according to claim 2, characterized in that, The step of outputting the position information of the target object in the target image based on the predicted position parameters corresponding to each position feature point in the positional convolutional feature map and the center position coordinates of the network pixel block corresponding to the position feature point includes: In the K grid pixels, the grid pixel block corresponding to the position feature point is obtained; the position feature point is obtained by convolution of the grid pixel block, and the size of the grid pixel block is determined by the downsampling factor corresponding to the K target pixel features; Obtain the coordinates of the center position of the grid pixel block; The location information of the target object is determined based on the predicted location parameters and the center location coordinates.

9. The method according to claim 8, characterized in that, The predicted position parameters include the position offset, the width of the predicted bounding box, and the height of the predicted bounding box; Determining the location information of the target object based on the predicted location parameters and the center location coordinates includes: Based on the position offset and the center position coordinates, the center position information corresponding to the prediction box is determined; Based on the center position information, the width of the prediction box, and the height of the prediction box, the position information of the prediction box in the target image is determined, and the position information of the prediction box in the target image is determined as the position information of the target object.

10. The method according to claim 1, characterized in that, Also includes: Obtain image samples, input the image samples into a sample image recognition model, and extract the sample features of the image samples in the sample image recognition model; The sample features are respectively input into the sample image classification component and the sample location prediction component of the sample image recognition model; the sample image classification component includes a first sample convolution parameter that focuses on object classification, and the sample location prediction component includes a second sample convolution parameter that focuses on location information prediction. In the sample image classification component, the sample features are convolved using the first sample convolution parameters to obtain a sample classification convolution feature map. In the sample location prediction component, the sample features are convolved using the second sample convolution parameters to obtain a sample location convolution feature map. Based on the sample classification convolutional feature map, the predicted object category of the sample object in the image sample is determined, and based on the sample position convolutional feature map, the predicted position information of the sample object is determined. Obtain the object category label of the sample object, obtain the location label information of the sample object, and determine the total loss value based on the predicted object category, the object category label, the predicted location information, and the location label information; The first sample convolution parameter and the second sample convolution parameter are adjusted according to the total loss value to obtain the first convolution parameter corresponding to the first sample convolution parameter and the second convolution parameter corresponding to the second sample convolution parameter. The sample image classification component including the first convolution parameter is determined as the image classification component, the sample location prediction component including the second convolution parameter is determined as the location prediction component, and the sample image recognition model including the image classification component and the location prediction component is determined as the image recognition model.

11. The method according to claim 10, characterized in that, The step of determining the total loss value based on the predicted object category, the object category label, the predicted location information, and the location label information includes: The classification loss value is determined based on the predicted object category and the object category label; The location loss value is determined based on the location label information and the predicted location information; The total loss value is generated based on the classification loss value and the location loss value.

12. The method according to claim 10, characterized in that, The step of adjusting the first sample convolution parameters and the second sample convolution parameters according to the total loss value to obtain the first convolution parameters corresponding to the first sample convolution parameters and the second convolution parameters corresponding to the second sample convolution parameters includes: A first partial derivative is determined between the total loss value and the predicted object category. The first sample convolution parameters are then adjusted based on the first partial derivative to obtain the first convolution parameters. A second partial derivative is determined between the total loss value and the predicted location information. The second sample convolution parameters are then adjusted based on the second partial derivative to obtain the second convolution parameters.

13. An image recognition device, characterized in that, include: The image acquisition module is used to acquire the target image and input the target image into the image recognition model; A feature extraction module is used to extract shared image features of the target image in the image recognition model; The feature input module is used to input the shared image features into the image classification component and the location prediction component in the image recognition model, respectively; the image classification component includes a first convolutional parameter that focuses on object classification, and the location prediction component includes a second convolutional parameter that focuses on location information prediction. The first convolutional parameter and the second convolutional parameter are independent of each other and have different focuses. The first feature convolution module is used in the image classification component to perform convolution processing on the shared image features through the first convolution parameters to obtain a classification convolution feature map. The classification convolution feature map contains multiple classification feature points. The output parameters corresponding to each classification feature point include the object existence probability and the category prediction probability. The object existence probability is the probability that a target object exists in the prediction box to which the classification feature point belongs. The prediction box is used to predict the position of the target object in the target image. The second feature convolution module is used in the location prediction component to perform convolution processing on the shared image features through the second convolution parameters to obtain a location convolution feature map. The location convolution feature map contains multiple location feature points. The output parameters corresponding to each location feature point include predicted location parameters. When the classification feature point and the location feature point correspond to the same network pixel block, one output parameter of the classification feature point corresponds to one output parameter of the location feature point. The category output module is used to output the object category of the target object in the target image based on the classification convolution feature map; The category output module, when outputting the object category of the target object in the target image based on the classification convolution feature map, is specifically used to output the object category of the prediction box to which each classification feature point belongs based on the object existence probability and category prediction probability corresponding to each classification feature point in the classification convolution feature map. The position output module is used to output the position information of the target object in the target image based on the predicted position parameters corresponding to each position feature point in the position convolution feature map and the center position coordinates of the network pixel block corresponding to the position feature point.

14. A computer device, characterized in that, include: Processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide network communication functions, the memory is used to store program code, and the processor is used to call the program code to execute the method according to any one of claims 1-12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions, which, when executed by a processor, perform the method according to any one of claims 1-12.

16. A computer program product comprising computer instructions, characterized in that, The computer instructions are stored in a computer-readable storage medium and, when read and executed by the processor of a computer device, cause the computer device to perform the method according to any one of claims 1-12.

Images