Image recognition method, device, computer equipment and computer readable storage medium

Abstract

The present invention discloses an image recognition method, device, computer equipment and computer-readable storage medium. The image recognition method comprises: obtaining an image to be recognized; locating a target object contained in the image to be recognized by using a pre-trained detection model, and determining the probability that the target object matches each classification in the detection model; and when at least one of the probabilities is within a preset range, extracting features of the target object by using a pre-trained feature model, and determining the final classification of the target object based on the extracted feature data. The image recognition method provided by the present invention can effectively improve the accuracy of image recognition.

Description

Background Art

[0002] With the birth and development of computer vision technology, machines can replace manual work to identify and classify objects in pictures, which can greatly save time and labor costs. However, at present, the probability of machine misidentification is still relatively high.

[0003] It should be noted that the above information disclosed in the background technology section is only used to enhance the understanding of the background of the present invention, and therefore it may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0004] In view of this, the present invention provides an image recognition method, apparatus, computer device and computer-readable storage medium.

[0005] Other features and advantages of the present invention will become apparent from the following detailed description, or may be learned in part by practice of the present invention.

[0006] According to one aspect of the present invention, there is provided an image recognition method, comprising: acquiring an image to be recognized; locating a target object contained in the image to be recognized through a pre-trained detection model, and determining the probability that the target object matches each classification in the detection model; and when at least one of the probabilities is within a preset range, extracting features of the target object through a pre-trained feature model, and determining a final classification of the target object based on the extracted feature data.

[0007] According to one embodiment of the present invention, the detection model is a detection model based on a two-stage model framework; determining the probability that the target object matches each classification in the detection model includes: inputting the located target object into a trained convolutional neural network for classification to determine the probability that the target object matches each classification in the detection model.

[0008] According to one embodiment of the present invention, the training samples of the convolutional neural network include new images obtained by performing at least one of the following operations on the original training image: flipping, rotating, blurring, scaling, cropping, color change, brightness change, contrast change, and saturation change.

[0009] According to one embodiment of the present invention, determining the final classification of the target object based on the extracted feature data includes: determining, in the feature model, a feature data cluster having the smallest Euclidean distance with the feature data set of the target object; and determining the classification corresponding to the feature data cluster as the final classification of the target object.

[0010] According to one embodiment of the present invention, each feature data cluster in the feature model is determined by pre-training multiple feature data sets of each classification based on a clustering algorithm; wherein the minimum Euclidean distance between each feature data cluster is greater than a preset first threshold, and the maximum Euclidean distance between feature data in each feature data cluster is less than a preset second threshold.

[0011] According to one embodiment of the present invention, the method further includes: based on the classification of the target object, retrieving and outputting the associated information of the classification in a pre-established knowledge graph.

[0012] According to an embodiment of the present invention, the association information of the categories includes: background information of the categories and relationship information between the categories.

[0013] According to another aspect of the present invention, there is provided an image recognition device, comprising: an image acquisition module, for acquiring an image to be recognized; a target detection module, for locating a target object contained in the image to be recognized through a pre-trained detection model, and determining the probability that the target object matches each classification in the detection model; and a classification determination module, for extracting features of the target object through a pre-trained feature model when at least one of the probabilities is within a preset range, and determining a final classification of the target object based on the extracted feature data.

[0014] According to another aspect of the present invention, a computer device is provided, comprising: a memory, a processor, and executable instructions stored in the memory and executable in the processor, wherein the processor implements any one of the above-mentioned image recognition methods when executing the executable instructions.

[0015] According to another aspect of the present invention, a computer-readable storage medium is provided, on which computer-executable instructions are stored, and when the executable instructions are executed by a processor, any of the above-mentioned image recognition methods is implemented.

[0016] The image recognition method provided by the present invention can effectively improve the accuracy of image recognition.

[0017] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present invention, and together with the specification are used to explain the principles of the present invention. Obviously, the accompanying drawings described below are only some embodiments of the present invention, and for those of ordinary skill in the art, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0019] Figure 1 is a flowchart of an image recognition method according to an exemplary embodiment.

[0020] Figure 2 is a flowchart showing another image recognition method according to an exemplary embodiment.

[0021] Figure 3 is a flowchart of yet another image recognition method according to an exemplary embodiment.

[0022] Figure 4 is a block diagram of an image recognition device according to an exemplary embodiment.

[0023] Figure 5 It is a schematic diagram showing the structure of a computer device according to an exemplary embodiment. DETAILED DESCRIPTION

[0024] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present invention will be more comprehensive and complete and fully convey the concepts of the example embodiments to those skilled in the art. The accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0025] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present invention. However, those skilled in the art will appreciate that the technical solution of the present invention may be practiced while omitting one or more of the specific details, or other methods, devices, steps, etc. may be adopted. In other cases, known structures, methods, devices, implementations or operations are not shown or described in detail to avoid obscuring various aspects of the present invention.

[0026] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0027] The solution provided by the present invention uses computer vision technology to replace manual image recognition. For ease of understanding, the concept of "computer vision" involved in the present invention is first explained below:

[0028] Computer Vision (CV) is a science that studies how to make machines "see". To be more specific, it refers to the use of cameras and computers to replace human eyes to identify, track and measure targets, and further process images so that the computer processes them into images that are more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D (3-Dimension) technology, virtual reality, augmented reality, synchronous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0029] As described above, the present invention proposes a new image recognition method, which will be specifically described below through various embodiments of the present invention.

[0030] Figure 1 FIG. 1 is a flow chart showing an image recognition method according to an exemplary embodiment. Figure 1 The image recognition method shown can be applied to the server side of an e-commerce supervision platform, for example.

[0031] refer to Figure 1 , the image recognition method 10 comprises:

[0032] In step S102, an image to be recognized is obtained.

[0033] In step S104, the target object contained in the image to be identified is located by using the pre-trained detection model, and the probability that the target object matches each classification in the detection model is determined.

[0034] In step S106, when at least one of the probabilities is within a preset range, features of the target object are extracted using a pre-trained feature model, and a final classification of the target object is determined based on the extracted feature data.

[0035] It should be noted that the present invention does not impose any limitation on the preset range, which can be any range in [0, 1], including [0, 1]. That is, step S106 can be performed in any case.

[0036] The image recognition method provided by the embodiment of the present invention can effectively improve the accuracy of image recognition.

[0037] It should be clearly understood that the present invention describes how to form and use specific examples, but the principles of the present invention are not limited to any details of these examples. On the contrary, based on the teachings of the content disclosed in the present invention, these principles can be applied to many other embodiments.

[0038] Based on the above, the present invention is described by taking the scenario of e-commerce supervision of animation IP (Intellectual Property) as an example. However, it should be noted that the method of the present invention is only described by way of example with respect to this scenario, and is not limited to this scenario and any concepts involved therein.

[0039] The purpose of e-commerce supervision of animation IP is to detect the products of merchants that have not obtained animation IP authorization, so as to remove them from the shelves, punish them or warn them accordingly. The platform server obtains several images uploaded by various merchants, which are taken by merchants from their online products (such as clothing, backpacks, dolls and other peripheral products, which may contain animation characters, i.e., the "target object" in step S104).

[0040] At this stage, the basic framework for realizing target detection mainly includes: (1) two-stage models represented by RCNN, Fast-CNN, Faster-CNN, Faster-RCNN, etc.; (2) one-stage models represented by YOLO, SSD, etc. Under the premise of ensuring the detection accuracy, the one-stage model can be selected to improve the real-time detection rate. In some embodiments of the present invention, the detection model can be a detection model based on the two-stage model (for example: Faster-RCNN) framework to balance various indicators and enhance generalization ability. Accordingly, step S104 may include: inputting the located target object into a trained convolutional neural network for classification to determine the probability that the target object matches each classification in the detection model.

[0041] For the detection scenario of animation IP, before training the detection model, the collected images need to be classified and sorted according to a classification label corresponding to each animation character (that is, the "classification" in the detection model is the character name), and the detection borders are annotated using annotation tools, and the output is, for example, an XML (Extensible Markup Language) file in VOC format as a carrier of the detection border position information. The images with labeled information are input into, for example, Faster-RCNN for supervised training, and a portion of the images are retained as a validation set to verify the model effect and determine the optimal model.

[0042] In some embodiments, the training samples of the convolutional neural network may include new images obtained by performing at least one of the following operations on the original training images: flipping, rotating, blurring, scaling, cropping, color change, brightness change, contrast change, and saturation change, that is, by taking a series of enhancement measures on the sample data to improve the generalization ability of the detection model application.

[0043] After the image to be identified passes through the detection model, due to the diversity of image features, the generalization ability of abstract features is too strong, and some target objects may be misdetected, that is, some target objects have a high probability of matching with different cartoon characters. To solve this problem, it is necessary to start with deep learning of character image features.

[0044] Figure 2 is a flowchart of another image recognition method according to an exemplary embodiment. Figure 1 The difference of the method 10 shown is that Figure 2 The method shown further provides a specific method for applying the feature model to determine the final classification of the target object, that is, further provides an embodiment of the above step S106. Figure 2 The image recognition method shown can also be applied to the server side of an e-commerce supervision platform, for example.

[0045] refer to Figure 2 , step S106 may include:

[0046] In step S1062 , in the feature model, a feature data cluster having the smallest Euclidean distance to the feature data set of the target object is determined.

[0047] In some embodiments, each feature data cluster in the feature model can be determined by pre-training multiple feature data sets of each classification based on a clustering algorithm, wherein the minimum Euclidean distance between each feature data cluster is greater than a preset first threshold, and the maximum Euclidean distance between feature data in each feature data cluster is less than a preset second threshold.

[0048] Based on the above, for example, feature extraction networks such as VGG16 and ResNet-50 can be used to extract features from images, and loss functions such as Triplet Loss, Center Loss, and Contractive Loss can be used to train feature data. In the training process of the feature model, the distance between sample data of the same category (animation characters) is made as small as possible, and the distance between sample data of different categories is made as large as possible, so that the final classification of the target object can be accurately divided during the application of the feature model. It should be noted that the above purpose can be achieved by adjusting the parameters in the loss function that control the degree of clustering of the features themselves during the training process.

[0049] In step S1064, the classification corresponding to the feature data cluster is determined as the final classification of the target object.

[0050] As mentioned above, some target objects may be misdetected only by the detection model. However, through deep learning of image features by the feature model, the possibility of mismatch between the shallow feature representation of the target object and the determined classification can be eliminated to the greatest extent.

[0051] The final classification result of the target object (i.e., whether the image contains a certain anime character) can be compared with the merchant's authorized IP list stored on the platform server to verify whether the merchant has infringed on anime IP rights. In addition, it can be used to provide consumers with rich product background information.

[0052] In accordance with the above, Figure 3 is a flowchart of another image recognition method according to an exemplary embodiment, which further provides an embodiment of the above method 10. Figure 3 The image recognition method shown can also be applied to the server side of an e-commerce supervision platform, for example.

[0053] refer to Figure 3 , method 10 may further include:

[0054] In step S108, according to the classification of the target object, the associated information of the classification is retrieved and output in the pre-established knowledge graph.

[0055] In some embodiments, the associated information of the categories may include: the background information of the categories and the relationship information between the categories. Still taking the animation IP as an example: when establishing the knowledge graph, the name of the animation to which the animation character belongs, the background information of the animation character, and the associated information such as the relationship between the animation characters can be written.

[0056] Those skilled in the art will appreciate that all or part of the steps for implementing the above embodiments are implemented as a computer program executed by a CPU. When the computer program is executed by the CPU, the above functions defined by the above method provided by the present invention are performed. The program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk.

[0057] In addition, it should be noted that the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0058] The following are embodiments of the device of the present invention, which can be used to implement the method embodiments of the present invention. For details not disclosed in the device embodiments of the present invention, please refer to the method embodiments of the present invention.

[0059] Figure 4 is a block diagram of an image recognition device according to an exemplary embodiment.

[0060] refer to Figure 4 The image recognition device 40 includes: an image acquisition module 402 , a target detection module 404 and a classification determination module 406 .

[0061] The image acquisition module 402 is used to acquire an image to be identified.

[0062] The target detection module 404 is used to locate the target object contained in the image to be identified by using a pre-trained detection model, and to determine the probability that the target object matches each classification in the detection model.

[0063] The classification determination module 406 is used to extract features of the target object through a pre-trained feature model when at least one of the probabilities is within a preset range, and determine the final classification of the target object based on the extracted feature data.

[0064] The image recognition device provided according to the embodiment of the present invention can effectively improve the accuracy of image recognition.

[0065] It should be noted that the block diagrams shown in the above figures are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0066] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0067] Figure 5 is a schematic diagram of a computer device according to an exemplary embodiment. It should be noted that: Figure 5 The computer device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0068] like Figure 5As shown, the computer device 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 are also stored. The CPU 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0069] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that a computer program read therefrom is installed into the storage section 808 as needed.

[0070] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 809, and / or installed from a removable medium 811. When the computer program is executed by a central processing unit (CPU) 801, the above-mentioned functions defined in the device of the present invention are executed.

[0071] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0072] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0073] The units involved in the embodiments of the present invention may be implemented in software or hardware. The units described may also be provided in a processor. For example, they may be described as follows: a processor includes a sending unit, an acquiring unit, a determining unit, and a first processing unit. The names of these units do not, in some cases, limit the units themselves. For example, the sending unit may also be described as a "unit that sends a request for obtaining an image to a connected server."

[0074] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes:

[0075] Acquire an image to be identified; locate a target object contained in the image to be identified through a pre-trained detection model, and determine the probability that the target object matches each classification in the detection model; and when at least one of the probabilities is within a preset range, extract features of the target object through a pre-trained feature model, and determine the final classification of the target object based on the extracted feature data.

[0076] The exemplary embodiments of the present invention are specifically shown and described above. It should be understood that the present invention is not limited to the detailed structure, configuration or implementation method described herein; on the contrary, the present invention is intended to cover various modifications and equivalent configurations included in the spirit and scope of the appended claims.

Claims

1. An image recognition method, characterized in that: include: Obtain an image to be recognized; Using a pre-trained detection model, locate the target object contained in the image to be identified, and determine the probability that the target object matches each classification in the detection model; and When at least one of the probabilities is within a preset range, features of the target object are extracted through a pre-trained feature model, and based on the extracted feature data, a feature data cluster with the smallest Euclidean distance to the feature data set of the target object is determined in the feature model to determine the final classification of the target object, and each feature data cluster in the feature model is determined by pre-training multiple feature data sets of each classification based on a clustering algorithm.

2. The method according to claim 1, characterized in that: The detection model is a detection model based on a two-stage model framework; Determining the probability that the target object matches each classification in the detection model includes: inputting the located target object into a trained convolutional neural network for classification to determine the probability that the target object matches each classification in the detection model.

3. The method according to claim 2, characterized in that The training samples of the convolutional neural network include new images obtained by performing at least one of the following operations on the original training image: flipping, rotating, blurring, scaling, cropping, color transformation, brightness transformation, contrast transformation, and saturation transformation.

4. The method according to any one of claims 1 to 3, characterized in that: Determining the final classification of the target object according to the extracted feature data includes: In the feature model, determining a feature data cluster having the smallest Euclidean distance to a feature data set of the target object; and The classification corresponding to the feature data cluster is determined as the final classification of the target object.

5. The method according to claim 4, characterized in that The minimum Euclidean distance between each feature data cluster is greater than a preset first threshold, and the maximum Euclidean distance between feature data in each feature data cluster is less than a preset second threshold.

6. The method according to any one of claims 1 to 3, characterized in that: Also includes: According to the classification of the target object, the associated information of the classification is retrieved and output in a pre-established knowledge graph.

7. The method according to claim 6, characterized in that The category association information includes: background information of the category and relationship information between categories.

8. An image recognition device, characterized in that: include: An image acquisition module, used for acquiring an image to be recognized; A target detection module, used to locate the target object contained in the image to be identified by using a pre-trained detection model, and determine the probability that the target object matches each classification in the detection model; and The classification determination module is used to extract features of the target object through a pre-trained feature model when at least one of the probabilities is within a preset range, and determine, based on the extracted feature data, a feature data cluster with the smallest Euclidean distance to the feature data set of the target object in the feature model to determine the final classification of the target object, wherein each feature data cluster in the feature model is determined by pre-training multiple feature data sets of each classification based on a clustering algorithm.

9. A computer device comprising: A memory, a processor, and executable instructions stored in the memory and executable in the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the executable instructions.

10. A computer-readable storage medium having computer-executable instructions stored thereon, characterized in that: When the executable instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images