Image-based subject recognition method and device, storage medium and electronic equipment

Abstract

The present disclosure relates to an image-based subject identification method and device, a storage medium and an electronic device. The method comprises: in response to a subject identification request, obtaining a combined feature of a to-be-identified image, the combined feature comprising a text feature for describing a subject object in the to-be-identified image and a feature set of each candidate object in the to-be-identified image; and according to the combined feature of the to-be-identified image, identifying the combined feature through a pre-trained decision tree to obtain an identification result output by the decision tree, the identification result being used to represent whether each candidate object is a subject object in the to-be-identified image or not. Since the decision tree has a lighter feature, the resource occupation is less, and TensorRT conversion is not required, and the task scenario with high request quantity can also be coped with.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of electronic information technology, and in particular, to a subject recognition method and device based on images, a storage medium and an electronic device. BACKGROUND

[0002] For an object retrieval system with a search function, it is necessary to accurately establish an object library. The object retrieval system queries information corresponding to a search request input by a user terminal from the object library and feeds back to the user terminal.

[0003] Among them, for the scenario of establishing an object library based on pictures, the picture may include both subject objects and non-subject objects, and it is necessary to accurately identify the subject objects of the picture and write the features of the subject objects into the object library, so as to accurately feed back the search results to the user terminal based on the object library in the subsequent. In some high-request object recognition task scenarios, it is necessary to accurately and quickly identify the subject. Therefore, how to accurately and quickly identify the subject when dealing with high-request object recognition tasks is a technical problem to be solved. SUMMARY

[0004] This summary is provided to introduce a selection of concepts, which will be described with greater specificity in the detailed description section. This summary does not intend to identify key or essential features of the claimed technology, nor is it intended for use in limiting the scope of the claimed technology.

[0005] In a first aspect, the present disclosure provides a subject recognition method based on images, comprising:

[0006] In response to a subject recognition request, a combined feature of a to-be-recognized image is obtained, the combined feature comprising a text feature for describing a subject object in the to-be-recognized image and a feature set of each candidate object in the to-be-recognized image;

[0007] According to the combined feature of the to-be-recognized image, the combined feature is identified by a pre-trained decision tree, and an identification result output by the decision tree is obtained, the identification result being used to represent whether each candidate object is a subject object in the to-be-recognized image or not.

[0008] In a second aspect, the present disclosure provides a subject recognition device based on images, comprising:

[0009] A first obtaining module is configured to, in response to a subject recognition request, obtain a combined feature of a to-be-recognized image, the combined feature comprising a text feature for describing a subject object in the to-be-recognized image and a feature set of each candidate object in the to-be-recognized image;

[0010] The recognition module is used to recognize the combined features of the image to be recognized by a pre-trained decision tree, and obtain the recognition result output by the decision tree. The recognition result is used to characterize whether each candidate object is a main object in the image to be recognized or not a main object in the image to be recognized.

[0011] Thirdly, this disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the subject identification method described in the first aspect.

[0012] Fourthly, this disclosure provides an electronic device, comprising:

[0013] A storage device on which computer programs are stored;

[0014] A processing device for executing the computer program in the storage device to implement the steps of the subject identification method according to any one of the first aspects.

[0015] The above technical solution responds to a subject recognition request and obtains combined features of the image to be recognized. These combined features include text features describing the main object in the image and feature sets of each candidate object in the image. Based on these combined features, a pre-trained decision tree is used to identify them, yielding a recognition result output by the decision tree. This recognition result characterizes whether each candidate object is the main object in the image or not. Because decision trees are lightweight, they consume fewer resources, do not require TensorRT conversion, and can handle high-request scenarios. Furthermore, since the combined features are discrete, and discrete features perform better in decision trees than in deep network models, using decision trees to process discrete features ensures the accuracy of subject recognition.

[0016] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0017] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale. In the drawings:

[0018] Figure 1 This is a flowchart illustrating an image-based subject recognition method according to an exemplary embodiment of the present disclosure.

[0019] Figure 2 This is a block diagram illustrating an image-based subject recognition device according to an exemplary embodiment of the present disclosure.

[0020] Figure 3 This is a schematic diagram of the structure of an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0021] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0022] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0023] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0024] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0025] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0026] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0027] As mentioned in the background section, images may contain both main and non-main objects. Therefore, it's crucial to accurately identify the main objects in an image and then build an object database based on their features. This database is then used to provide accurate search results to the user's terminal. Currently, most methods employ deep network-based graph convolutional mapping (GCM) subject recognition models to identify subjects in images. However, to make these models suitable for high-demand subject recognition tasks, they need to be converted and deployed using TensorRT to improve computational speed. However, because the library files used during training in these models are from third-party libraries supported by TensorRT, which TensorRT cannot support, the deep network-based GCM subject recognition models cannot be converted and deployed using TensorRT. Consequently, these models are unsuitable for high-demand tasks.

[0028] In view of this, the present disclosure provides an image-based subject recognition method, apparatus, storage medium, and electronic device, and proposes a subject recognition scheme suitable for high-request-volume task scenarios, solving the problem that graph convolutional subject recognition models based on deep networks cannot be applied to high-request-volume task scenarios.

[0029] The embodiments of this disclosure will be further explained below with reference to the accompanying drawings.

[0030] Figure 1 This disclosure illustrates an image-based subject recognition method according to an exemplary embodiment, which can be applied to electronic devices. (Refer to...) Figure 1 The subject identification method includes the following steps:

[0031] Step S101: In response to the subject recognition request, obtain the combined features of the image to be recognized. The combined features include text features describing the subject object in the image to be recognized and feature sets of each candidate object in the image to be recognized.

[0032] It should be noted that text features are used to describe the main objects in the image to be identified. The feature set of each candidate object in the image to be identified is a set of features describing each candidate object.

[0033] In some embodiments, the text features of the main object in the image to be identified can be obtained by describing the text of the image to be identified, and the feature sets of each candidate object in the image to be identified can be obtained by performing object detection on the image to be identified. In some embodiments, the text features and the feature sets of each candidate object can be obtained by using a deep model. In this case, step S101 may include: responding to a subject recognition request, obtaining the image to be identified and the text corresponding to the image to be identified; determining the text features used to describe the main object in the image to be identified based on the text; performing object detection on the image to be identified based on the image to be identified using an object detection model to obtain object detection results, the object detection results including the position information of each candidate detection box in the image to be identified and the category label of the candidate object corresponding to each candidate detection box; determining the feature sets of each candidate object in the image to be identified based on the object detection results; and determining the combined features of the image to be identified based on the text features and the feature sets of each candidate object.

[0034] In this embodiment, text features can be extracted from the text using a text feature extraction model. This model can be used to extract keywords. It is understood that the text describing the image to be identified generally carries keyword information, which can be used to describe the main object in the image. Therefore, the text feature extraction model can be used to extract keywords from the text as text features of the image to be identified. For example, if the text contains words such as "clothing," it can indicate that the image to be identified may be an image displaying clothing, and the clothing object can be considered the main object in the image.

[0035] In this embodiment, object detection, which involves finding all objects of interest in an image and determining their location and category, is one of the core problems in the field of computer vision. In this embodiment, an object detection model is used to perform object detection, which can detect bounding boxes containing objects and the category corresponding to the objects contained within those bounding boxes. In some embodiments, the object detection model can be a neural network model based on Faster R-CNN, SSD, and YOLO; this embodiment does not limit this approach. Furthermore, the object detection model trained using neural network models based on Faster R-CNN, SSD, and YOLO can be referenced from related technologies, which will not be elaborated upon in this embodiment.

[0036] In some embodiments, for a feature set of a candidate object, the feature set of the candidate object may include the ontological features of the candidate object itself. The ontological features may, for example, be the position information and category information of the detection box corresponding to the candidate object. In this case, the step of determining the feature set for describing each candidate object in the image to be identified based on the target detection result may include: for each candidate detection box, determining the ontological features of the candidate object corresponding to the candidate detection box based on the position information of the candidate detection box and the category label of the candidate object corresponding to the candidate detection box.

[0037] In some embodiments, category information may include primary category label information and secondary category label information. For example, if the primary category label information is "high heels," the secondary category label information may be the heel height. As another example, if the primary category label information is "clothing," the secondary category label information may be "tops."

[0038] In some embodiments, the ontological features of a candidate object may further include the visual features corresponding to the candidate detection box. Specifically, the overall visual features of the image to be identified can be extracted based on a ResNet-50 network, and the visual features corresponding to each candidate detection box can be extracted based on the position information of each candidate detection box.

[0039] In some embodiments, the ontological features of a candidate object may further include a confidence score. The confidence score can be used to characterize the probability that an object exists in the candidate detection box corresponding to the candidate object, and it can also be used to characterize the probability corresponding to the candidate object category label output by the object detection model. In this case, the step of determining the feature set for describing each candidate object in the image to be identified based on the object detection result may include: for each candidate detection box, determining the ontological features of the candidate object corresponding to the candidate detection box based on the position information of the candidate detection box, the category label of the candidate object corresponding to the candidate detection box, and the confidence score corresponding to the candidate detection box.

[0040] In some embodiments, for a feature set of a candidate object, the feature set of the candidate object may further include the association features between the candidate object and other candidate objects. These association features characterize the correlation between the candidate object and other candidate objects. In this case, the step of determining the feature set describing each candidate object in the image to be identified based on the target detection result may include: for each candidate detection box, determining the similarity features between the candidate object in the candidate detection box and the candidate objects in other candidate detection boxes, and determining the association features of the candidate object corresponding to the candidate detection box based on the similarity features.

[0041] To determine the association features of candidate objects in candidate detection box A, taking candidate detection boxes B and C as examples, the similarity feature of candidate objects in candidate detection box A can be, for example, the sum of the similarities between candidate detection boxes A and B and between candidate detection boxes A and C; or, for example, the maximum value among the similarities between candidate detection boxes A and B and between candidate detection boxes A and C; or, for example, the mean among the similarities between candidate detection boxes A and B and between candidate detection boxes A and C. The similarity between two candidate detection boxes can be calculated based on their visual features. In this embodiment, for ease of comparison, different similarity features can be normalized to map to the same dimension.

[0042] In some embodiments, the images to be identified may include multiple images, and the multiple images to be identified highlight the same subject object. In this case, the feature set of the candidate object may also include the probability of objects of the same category and the probability of objects of the same type. The probability of objects of the same category of the candidate object is used to characterize the probability of the category label corresponding to the candidate object appearing in the target detection results of all images to be identified, and the probability of objects of the same type of the candidate object is used to characterize the probability of the candidate object appearing in all images to be identified.

[0043] In this embodiment, if the similarity between two candidate detection boxes representing the same object is greater than a preset similarity threshold, then the candidate objects in the two candidate detection boxes can be considered to belong to the same object. The similarity between two candidate detection boxes can be calculated using the visual features corresponding to the two candidate detection boxes. The preset similarity threshold can be 0.7 or other values, which are not limited in this embodiment.

[0044] For example, taking three images to be identified, the first image contains three candidate detection boxes (candidate detection box A1, candidate detection box A2, and candidate detection box A3), the second image contains three candidate detection boxes (candidate detection box B1 and candidate detection box B2), and the third image contains three candidate detection boxes (candidate detection box C1 and candidate detection box C2). The category labels of the candidate objects corresponding to candidate detection boxes A1, B1, and C1 are all clothing labels, the category label of the candidate object corresponding to candidate detection box A2 is a shoe label, the category labels of the candidate objects corresponding to candidate detection boxes A3 and B2 are both bag labels, and the category label of the candidate object corresponding to candidate detection box C2 is a jewelry label. Furthermore, the similarity between the candidate objects corresponding to candidate detection boxes A1 and C1 is greater than 0.7. Taking the calculation of the probability of a candidate object in candidate detection box A1 being of the same category and the probability of a candidate object being the same style as another candidate object as yet ...

[0045] In this embodiment, combining multiple images to be identified for subject recognition can improve the accuracy of the image subject recognition method.

[0046] Step S102: Based on the combined features of the image to be identified, the combined features are identified by a pre-trained decision tree to obtain the identification result output by the decision tree. The identification result is used to characterize whether each candidate object is the main object in the image to be identified or not.

[0047] In some embodiments, the recognition result may be the probability that each candidate object in the image to be recognized belongs to the main object. Based on the magnitude of the probability, it can be determined whether each candidate object is the main object in the image to be recognized.

[0048] In some embodiments, the decision tree can be an XGBoost (Extreme Gradient Boosting) model, a random forest model, etc., which will not be elaborated here. Specifically, for the XGBoost model, the XGBoost model is trained using an ensemble algorithm. The XGBoost model includes multiple trees, and the results of these multiple trees are combined to generate the final recognition result.

[0049] The following explanation uses a decision tree as the XGBoost model to illustrate the training process of the decision tree in this disclosure. The XGBoost model can be trained as follows: The sample combination features of each sample image in the sample image set and the labels of each candidate detection box in each sample image are obtained. The sample combination features of the sample images include the text features of the sample images and the feature sets of each candidate object in the sample images. Based on the sample combination features of each sample image in the sample image set and the labels of each candidate detection box in each sample image, the XGBoost model is trained to obtain the trained XGBoost model. The specific training process can be referenced from related technologies, and will not be elaborated upon in this embodiment.

[0050] The acquisition of text features of sample images and feature sets of each candidate object in sample images can be referred to the relevant content in step S101, which will not be repeated here in this embodiment.

[0051] In each sample image, the label of each candidate detection box is used to indicate whether the candidate object corresponding to the candidate detection box is the main object.

[0052] Among these advantages, the XGBoost model offers several benefits. For instance, before training, the XGBoost model pre-sorts the sample combination features and stores them in memory as blocks, allowing for the reuse of these features during iterative training. This block structure also enables parallel computation; when splitting a node, the gain of each feature needs to be calculated, and the feature with the highest gain is selected for splitting. This gain calculation can be performed in parallel using multiple threads, thus saving training time. Furthermore, since the features used for subject recognition (text features and feature sets) are discrete, and the XGBoost model is well-suited for handling discrete features, its use ensures high accuracy in subject recognition.

[0053] In some embodiments, after obtaining the recognition result output by the decision tree, the main object (e.g., a product) in the image to be recognized can be determined. Based on the features of the main object, an object database is established, which may include the features of the main object and information corresponding to those features (e.g., product description links, purchase links). Therefore, in a search scenario based on the object database, the image to be recognized can be a frame from a video played on a user terminal. Recognition can be performed on a frame from the video. If the main object in the image is identified, features matching the features of the main object in the image can be retrieved from the object database, and the information corresponding to those features can be fed back to the user terminal. This information may include product description links, purchase links, etc.

[0054] In this way, decision trees are more lightweight than deep network models, thus requiring less resources and eliminating the need for TensorRT transformation, making them suitable for high-demand tasks. Furthermore, since the combined features are discrete features, and discrete features perform better in decision trees than in deep network models, using decision trees to process discrete features ensures the accuracy of subject identification.

[0055] Figure 2 This is an exemplary embodiment of the present disclosure illustrating an image-based subject recognition device, the subject recognition device 200 comprising:

[0056] The first acquisition module 201 is used to respond to a subject recognition request and acquire combined features of the image to be recognized. The combined features include text features describing the subject object in the image to be recognized and feature sets of each candidate object in the image to be recognized.

[0057] The recognition module 202 is used to recognize the combined features of the image to be recognized by a pre-trained decision tree, and obtain the recognition result output by the decision tree. The recognition result is used to characterize whether each candidate object is a main object in the image to be recognized or not a main object in the image to be recognized.

[0058] Optionally, the first acquisition module 201 includes:

[0059] The first acquisition submodule is used to respond to a subject recognition request and acquire the image to be recognized and the text corresponding to the image to be recognized;

[0060] The text feature determination submodule is used to determine text features that describe the main object in the image to be identified based on the text.

[0061] The target detection submodule is used to perform target detection on the image to be identified using a target detection model to obtain target detection results. The target detection results include the position information of each candidate detection box in the image to be identified and the category label of the candidate object corresponding to each candidate detection box.

[0062] The first determining submodule is used to determine the feature set of each candidate object in the image to be identified based on the target detection result;

[0063] The second determining submodule is used to determine the combined features of the image to be identified based on the text features and the feature sets of each of the candidate objects.

[0064] Optionally, the feature set of the candidate object includes the ontological features of the candidate object, and the first determining submodule includes:

[0065] The first determining unit is configured to, for each candidate detection box, determine the ontological features of the candidate object corresponding to the candidate detection box based on the position information of the candidate detection box and the category label of the candidate object corresponding to the candidate detection box.

[0066] Optionally, the feature set of the candidate object further includes the association features of the candidate object, which are used to characterize the correlation between the candidate object and other candidate objects besides the candidate object. The first determining submodule further includes:

[0067] The second determining unit is configured to, for each candidate detection box, determine the similarity features between the candidate object in the candidate detection box and the candidate objects in other candidate detection boxes besides the candidate detection box; and determine the association features of the candidate object corresponding to the candidate detection box based on the similarity features.

[0068] Optionally, the images to be identified include multiple images, and the multiple images to be identified highlight the same subject object. The feature set of the candidate object also includes the probability of objects of the same category and the probability of objects of the same type. The probability of objects of the same category of the candidate object is used to characterize the probability of the category label corresponding to the candidate object appearing in the target detection results corresponding to all the images to be identified. The probability of objects of the same type of the candidate object is used to characterize the probability of the candidate object appearing in all the images to be identified.

[0069] Optionally, the decision tree is an XGBoost model.

[0070] The following is for reference. Figure 3This diagram illustrates a structural schematic of an electronic device 300 suitable for implementing embodiments of the present disclosure. The terminal devices in these embodiments may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0071] like Figure 3 As shown, the electronic device 300 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The processing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0072] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0073] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 309, or installed from storage device 308, or installed from ROM 302. When the computer program is executed by processing device 301, it performs the functions defined in the methods of embodiments of this disclosure.

[0074] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0075] In some implementations, electronic devices can communicate using any currently known or future-developed network protocol, such as HTTP (Hypertext Transfer Protocol), and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0076] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0077] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: respond to a subject recognition request, acquire combined features of an image to be recognized, the combined features including text features of the image to be recognized and feature sets of each candidate object in the image to be recognized; and, based on the combined features of the image to be recognized, identify the combined features using a pre-trained decision tree to obtain a recognition result output by the decision tree, the recognition result being used to characterize each candidate object as a subject object in the image to be recognized or not as a subject object in the image to be recognized.

[0078] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0079] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0080] The modules described in the embodiments of this disclosure can be implemented in software or in hardware. The names of the modules are not necessarily limiting in certain circumstances; for example, the first acquisition module can also be described as "a module for acquiring combined features of an image to be identified".

[0081] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0082] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0083] According to one or more embodiments of this disclosure, Example 1 provides an image-based subject recognition method, the subject recognition method comprising:

[0084] In response to a subject recognition request, the system obtains combined features of the image to be recognized, including text features describing the subject object in the image to be recognized and feature sets of each candidate object in the image to be recognized.

[0085] Based on the combined features of the image to be identified, the combined features are identified by a pre-trained decision tree to obtain the identification result output by the decision tree. The identification result is used to characterize whether each candidate object is the main object in the image to be identified or is not the main object in the image to be identified.

[0086] According to one or more embodiments of this disclosure, Example 2 provides a subject recognition method of Example 1, wherein the method of obtaining combined features of an image to be recognized in response to a subject recognition request includes:

[0087] In response to a subject recognition request, obtain the image to be recognized and the text corresponding to the image to be recognized;

[0088] Based on the text, determine the text features used to describe the main object in the image to be identified;

[0089] Based on the image to be identified, target detection is performed on the image to be identified using a target detection model to obtain target detection results. The target detection results include the position information of each candidate detection box in the image to be identified and the category label of the candidate object corresponding to each candidate detection box.

[0090] Based on the target detection results, the feature set of each candidate object in the image to be identified is determined;

[0091] Based on the text features and the feature sets of each candidate object, the combined features of the image to be identified are determined.

[0092] According to one or more embodiments of this disclosure, Example 3 provides the subject recognition method of Example 2, wherein the feature set of the candidate object includes the ontological features of the candidate object, and the step of determining the feature set for describing each candidate object in the image to be recognized based on the target detection result includes:

[0093] For each candidate detection box, the ontological features of the candidate object corresponding to the candidate detection box are determined based on the position information of the candidate detection box and the category label of the candidate object corresponding to the candidate detection box.

[0094] According to one or more embodiments of this disclosure, Example 4 provides the subject recognition method of Example 2, wherein the feature set of the candidate object further includes the association features of the candidate object, the association features of the candidate object being used to characterize the correlation between the candidate object and other candidate objects besides the candidate object, and the step of determining the feature set used to describe each candidate object in the image to be recognized based on the target detection result further includes:

[0095] For each candidate detection box, determine the similarity features between the candidate object in that candidate detection box and the candidate objects in other candidate detection boxes; and,

[0096] Based on the similarity features, the association features of the candidate objects corresponding to the candidate detection box are determined.

[0097] According to one or more embodiments of this disclosure, Example 5 provides the subject recognition method of Example 3, wherein the image to be recognized includes multiple images, and the multiple images to be recognized highlight the same subject object. The feature set of the candidate object also includes the probability of objects of the same category and the probability of objects of the same type. The probability of objects of the same category of the candidate object is used to characterize the probability of the category label corresponding to the candidate object appearing in the target detection results corresponding to all the images to be recognized. The probability of objects of the same type of the candidate object is used to characterize the probability of the candidate object appearing in all the images to be recognized.

[0098] According to one or more embodiments of this disclosure, Example 6 provides a subject identification method of any one of Examples 1-5, wherein the decision tree is an XGBoost model.

[0099] According to one or more embodiments of this disclosure, Example 7 provides an image-based subject recognition device, the subject recognition device comprising:

[0100] The first acquisition module is used to respond to a subject recognition request and acquire combined features of the image to be recognized. The combined features include text features describing the subject object in the image to be recognized and feature sets of each candidate object in the image to be recognized.

[0101] The recognition module is used to recognize the combined features of the image to be recognized by a pre-trained decision tree, and obtain the recognition result output by the decision tree. The recognition result is used to characterize whether each candidate object is a main object in the image to be recognized or not a main object in the image to be recognized.

[0102] According to one or more embodiments of this disclosure, Example 8 provides the apparatus of Example 7, wherein the first acquisition module includes:

[0103] The first acquisition submodule is used to respond to a subject recognition request and acquire the image to be recognized and the text corresponding to the image to be recognized;

[0104] The text feature determination submodule is used to determine text features that describe the main object in the image to be identified based on the text.

[0105] The target detection submodule is used to perform target detection on the image to be identified using a target detection model to obtain target detection results. The target detection results include the position information of each candidate detection box in the image to be identified and the category label of the candidate object corresponding to each candidate detection box.

[0106] The first determining submodule is used to determine the feature set of each candidate object in the image to be identified based on the target detection result;

[0107] The second determining submodule is used to determine the combined features of the image to be identified based on the text features and the feature sets of each of the candidate objects.

[0108] According to one or more embodiments of the present disclosure, Example 9 provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, performs the steps of the subject identification method described in any one of Examples 1-6.

[0109] According to one or more embodiments of this disclosure, Example 10 provides an electronic device, including:

[0110] A storage device on which computer programs are stored;

[0111] A processing device for executing the computer program in the storage device to implement the steps of any of the subject identification methods in Examples 1-6.

[0112] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0113] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0114] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative forms of implementing the claims. Regarding the apparatus in the above embodiments, the specific manner in which the various modules perform their operations has been described in detail in the embodiments relating to the method, and will not be elaborated upon here.

Claims

1. An image-based subject recognition method, characterized in that, The subject identification method includes: In response to a subject recognition request, the system obtains combined features of the image to be recognized, including text features describing the subject object in the image to be recognized and feature sets of each candidate object in the image to be recognized. Based on the combined features of the image to be identified, the combined features are identified by a pre-trained decision tree to obtain the identification result output by the decision tree. The identification result is used to characterize whether each candidate object is the main object in the image to be identified or is not the main object in the image to be identified. The step of responding to a subject recognition request and obtaining combined features of the image to be recognized includes: responding to a subject recognition request and obtaining the image to be recognized and text corresponding to the image to be recognized; determining text features for describing the subject object in the image to be recognized based on the text; performing object detection on the image to be recognized using an object detection model to obtain object detection results, the object detection results including the position information of each candidate detection box in the image to be recognized and the category label of the candidate object corresponding to each candidate detection box; determining the feature set of each candidate object in the image to be recognized based on the object detection results; and determining the combined features of the image to be recognized based on the text features and the feature sets of each candidate object.

2. The subject identification method according to claim 1, characterized in that, The feature set of the candidate object includes the ontological features of the candidate object. The step of determining the feature set used to describe each candidate object in the image to be identified based on the target detection result includes: For each candidate detection box, the ontological features of the candidate object corresponding to the candidate detection box are determined based on the position information of the candidate detection box and the category label of the candidate object corresponding to the candidate detection box.

3. The subject identification method according to claim 1, characterized in that, The feature set of the candidate object also includes the association features of the candidate object, which are used to characterize the correlation between the candidate object and other candidate objects. The step of determining the feature set used to describe each candidate object in the image to be identified based on the target detection result further includes: For each candidate detection box, determine the similarity features between the candidate object in that candidate detection box and the candidate objects in other candidate detection boxes; and, Based on the similarity features, the association features of the candidate objects corresponding to the candidate detection box are determined.

4. The subject identification method according to claim 2, characterized in that, The images to be identified include multiple images, and the multiple images to be identified highlight the same subject object. The feature set of the candidate object also includes the probability of objects of the same category and the probability of objects of the same type. The probability of objects of the same category of the candidate object is used to characterize the probability of the category label corresponding to the candidate object appearing in the target detection results of all the images to be identified. The probability of objects of the same type of the candidate object is used to characterize the probability of the candidate object appearing in all the images to be identified.

5. The subject identification method according to any one of claims 1-4, characterized in that, The decision tree is an XGBoost model.

6. An image-based subject recognition device, characterized in that, The subject identification device includes: The first acquisition module is used to respond to a subject recognition request and acquire combined features of the image to be recognized. The combined features include text features describing the subject object in the image to be recognized and feature sets of each candidate object in the image to be recognized. The recognition module is used to identify the combined features of the image to be recognized by a pre-trained decision tree, and obtain the recognition result output by the decision tree. The recognition result is used to characterize whether each candidate object is the main object in the image to be recognized or is not the main object in the image to be recognized. The first acquisition module includes: The first acquisition submodule is used to respond to a subject recognition request and acquire the image to be recognized and the text corresponding to the image to be recognized; The text feature determination submodule is used to determine text features that describe the main object in the image to be identified based on the text. The target detection submodule is used to perform target detection on the image to be identified using a target detection model to obtain target detection results. The target detection results include the position information of each candidate detection box in the image to be identified and the category label of the candidate object corresponding to each candidate detection box. The first determining submodule is used to determine the feature set of each candidate object in the image to be identified based on the target detection result; The second determining submodule is used to determine the combined features of the image to be identified based on the text features and the feature sets of each of the candidate objects.

7. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processing device, it implements the steps of the subject identification method according to any one of claims 1-5.

8. An electronic device, characterized in that, include: A storage device on which computer programs are stored; A processing device for executing the computer program in the storage device to implement the steps of the subject identification method according to any one of claims 1-5.

Images