Course domain diagram object detection method, system, device and storage medium

Abstract

The present invention discloses a method, system, device and storage medium for detecting objects in a course domain diagram, belonging to the field of image target detection technology. The object detection method comprises extracting global feature information of an image and local feature information of an object to be detected and fusing them to obtain fused feature information; obtaining a predicted probability corresponding to the fused feature information, forming a key-value pair with the local feature information of the object to be detected and the predicted probability; writing the key-value pair into a memory network, updating the key-value pair and training an object detection network model by transferring a loss function, inputting a course domain diagram into the trained object detection network model, and outputting the objects contained in the course domain diagram. The present invention can solve the problem of large differences in visual feature information of course domain diagrams and a high scarcity of samples, enrich feature information, enhance the visual information representation of local feature information, and improve the detection performance of objects in course domain diagrams.

Description

Technical Field

[0001] The present invention belongs to the technical field of image target detection, and in particular relates to a method, system, device and storage medium for detecting objects in course field diagrams. Background Art

[0002] With the development of the information age, online knowledge resources have exploded, with an increasing abundance of learning materials, technical documentation, and instructional videos. These include human-created course domain diagrams. Solving the task of object detection on course domain diagrams, which can be used to determine the location information of various objects on the diagram, is a crucial foundation for cross-media, knowledge-intensive tasks such as knowledge fusion and intelligent question answering, and is a necessary step in promoting the further development of intelligent education.

[0003] However, current schematic research focuses primarily on architectural drawings, sketches, and hand-drawn images, with little research on schematics in the curriculum domain. Furthermore, current mainstream object detection models struggle to adapt to object detection tasks in curriculum diagrams due to the large disparity in visual feature information and the scarcity of samples. Summary of the Invention

[0004] The purpose of the present invention is to address the problems in the above-mentioned prior art and provide a method, system, device and storage medium for object detection in course field diagrams, so as to address the difficulties of large differences in visual feature information and high scarcity of samples in course field diagrams, and reliably realize target detection of objects contained in course field diagrams.

[0005] In order to achieve the above object, the present invention has the following technical solutions:

[0006] A method for detecting objects in a course domain diagram, comprising:

[0007] Extract the global feature information of the image and the local feature information of the object to be detected and fuse them to obtain fused feature information;

[0008] Obtain the predicted probability corresponding to the fused feature information, and form a key-value pair with the local feature information of the object to be detected and the predicted probability;

[0009] Write the key-value pairs into the memory network, update the key-value pairs and train the object detection network model by transferring the loss function, input the course domain diagram into the trained object detection network model, and output the objects contained in the course domain diagram.

[0010] As a preferred solution, the object detection network model adopts the Faster R-CNN network model.

[0011] As a preferred solution, the step of extracting global feature information of the image and local feature information of the object to be detected and fusing them to obtain fused feature information includes:

[0012] Use the pre-trained Faster R-CNN network model to obtain the global feature information s of the image;

[0013] Use the pre-trained Faster R-CNN network model to detect objects in the course domain image and obtain the local feature information of n objects {o1,o2,…,o n};

[0014] Combine the global feature information s of the image with the local feature information of n objects {o1, o2, ..., o n} to fuse and obtain the fusion feature information {v1,v2,…,v n}.

[0015] As a preferred solution, in the step of obtaining the prediction probability corresponding to the fused feature information, the obtained fused feature information {v1, v2, ..., v n} Input the pre-trained Faster R-CNN network model to obtain the corresponding prediction probability {p1,p2,…,p n}.

[0016] As a preferred solution, in the step of writing the key-value pair into the memory network, a Memory Bank module based on feature information and prediction probability is provided in the memory network, and the expression of the key-value pair is (K L ,V L ), where K and V are used to store feature information and prediction probability respectively, and L represents the number of object categories;

[0017] In the step of forming a key-value pair of the local feature information of the object to be detected and the predicted probability, the local feature information of the object {o1, o2, ..., o n} and predicted probabilities {p1,p2,…,p n}, the corresponding feature information and prediction probability form key-value pairs {(k1,l1),(k2,l2),...,(k n ,l n )}, the generated key-value pairs are passed to the Memory Bank module as the input of the memory network.

[0018] As a preferred solution, in the step of writing the key-value pair into the memory network, each time new feature information and predicted probability are obtained, it is determined whether the Memory Bank module of the current category is full. If it is not full, the memory writing phase is executed; if it is full, the memory updating phase is executed; in the memory writing phase, the newly obtained feature information and predicted probability key-value pair (k i ,l i ) is added to the end of the Memory Bank module.

[0019] As a preferred solution, in the step of updating the key-value pairs and training the object detection network model by transferring the loss function, the feature information with the highest similarity in the current Memory Bank module is selected for fusion. The calculation expressions of the similarity calculation method and the feature information fusion method are as follows:

[0020]

[0021]

[0022] In the formula, o represents the local feature information of the object, k represents the feature information stored in the current Memory Bank module, δ is used to ensure that the denominator is not 0, k neq Represents the new feature information after fusion, k maxsimilarity Indicates the feature information in the current MemoryBank module that is most similar to the newly acquired feature information;

[0023] Each time new feature information is learned, the corresponding Memory Bank module also generates a probability prediction for the current feature information. The calculation expression is as follows:

[0024]

[0025] Ideally, The value of is 1, and the loss function of the memory network is as follows:

[0026]

[0027] A course domain diagram object detection system, comprising:

[0028] The feature fusion module is used to extract the global feature information of the image and the local feature information of the object to be detected and fuse them to obtain fused feature information;

[0029] A key-value pair acquisition module is used to obtain the prediction probability corresponding to the fused feature information, and to form a key-value pair with the local feature information of the object to be detected and the prediction probability;

[0030] The model training module is used to write key-value pairs into the memory network, update the key-value pairs, and train the object detection network model by transferring the loss function;

[0031] The object output module is used to input the course domain diagram into the trained object detection network model and output the objects contained in the course domain diagram.

[0032] An electronic device, comprising:

[0033] a memory storing at least one instruction; and

[0034] The processor executes the instructions stored in the memory to implement the object detection method of the course domain diagram.

[0035] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for detecting objects in a course domain diagram is implemented.

[0036] Compared with the prior art, the present invention has at least the following beneficial effects:

[0037] To address the issues of large differences in visual feature information and a high scarcity of samples in course domain diagrams, the method of the present invention addresses the two main differences between course domain diagrams and natural scene images at the feature information level and sample quantity level. By fusing the global feature information of the diagram image with the local feature information of the object to be detected, the feature information is enriched and the visual information representation of the local feature information is enhanced. By memorizing the learned feature information and the predicted probability into key-value pairs, and continuously updating them in the subsequent learning process using a memory network, the training of the object detection network model is guided by the transfer loss function, thereby storing the diagram feature information and improving the detection performance of objects in course domain diagrams. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only show some embodiments of the present invention. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 is a flow chart of a method for detecting objects in a course domain diagram according to an embodiment of the present invention;

[0040] Figure 2 It is a model structure diagram of the object detection method of the course field schematic diagram according to an embodiment of the present invention;

[0041] Figure 3This is a structural diagram of a Memory Bank module according to an embodiment of the present invention;

[0042] Figure 4 is a probability calculation diagram of the Memory Bank module according to an embodiment of the present invention;

[0043] Figure 5 This is the experimental result of the model after adding the information fusion module on the course domain diagram in the embodiment of the present invention;

[0044] Figure 6 This is the experimental result of the model after adding the memory network module on the course domain diagram according to the embodiment of the present invention;

[0045] Figure 7 This is the experimental result of the model after adding information fusion and memory network on the course domain diagram according to the embodiment of the present invention;

[0046] Figure 8 The experimental results of the target detection task of different models according to the embodiment of the present invention on the course domain diagram dataset are shown;

[0047] Figure 9 These are the experimental results of the target detection task on the course domain diagram dataset using different memory slot parameters in the memory network of an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, ordinary technicians in this field can also derive other embodiments without making any creative work.

[0049] The object detection method of the course domain diagram in the embodiment of the present invention mainly includes three stages: 1) image feature extraction and fusion stage, 2) feature information classification stage, and 3) memory network writing, updating and transmission stage.

[0050] Figure 1 The flow chart of the object detection method in the course domain diagram according to the embodiment of the present invention is shown. Figure 2 The model structure of the object detection method for the course domain diagram according to an embodiment of the present invention is shown, which specifically includes the following steps:

[0051] 1) In the image extraction and fusion stage, first, the feature extraction module in the pre-trained object detection network Faster R-CNN model is used to obtain the global feature information s of the image; secondly, the pre-trained object detection network Faster R-CNN is used to detect the objects in the course domain image, and the positions of n objects in the image and the local feature information of the objects {o1, o2, ..., on}; Finally, the feature information of the first two steps is fused to obtain the new feature information {v1,v2,…,v n}, as shown in expression (1), and then the feature information {v1,v2,…,v n}Send to the feature information classification stage.

[0052] v i =Concat(o i ,s) (1)

[0053] 2) In the feature information classification stage, first, the feature information {v1, v2, ..., v n} is fed into the classifier module in the pre-trained object detection network Faster R-CNN model to obtain the corresponding prediction probabilities {p1,p2,…,p n}, and then the local feature information and its corresponding prediction probability are combined into a key-value pair for storage. When generating the key-value pair, the embodiment of the present invention considers avoiding the mutual influence of feature information between different categories and designs a Memory Bank mechanism based on feature information and prediction probability, which is denoted as (K L , V L ),like Figure 3 As shown in the figure, K and V are used to store feature information and prediction probability respectively, and L represents the number of object categories. n} and predicted probabilities {p1, p2, ..., p n}, the corresponding feature information and prediction probability form key-value pairs {(k1, l1), (k2, l2), ..., (k n , l n )}, the generated key-value pairs will be passed to the Memory Bank module as the input of the memory network.

[0054] 3) In the writing, updating and transmission phase of the memory network, first, the writing method is selected according to whether the current corresponding Memory Bank module is full. Every time new feature information and predicted probability are obtained, it is necessary to determine whether the Memory Bank module of the current category is full. If it is not full, the memory writing phase is executed. If it is full, the memory updating phase is executed. The memory writing phase directly writes the newly acquired feature information and predicted probability key-value pairs (k i , l i ) is added to the end of the Memory Bank module, as shown in Expression 2.

[0055]

[0056] The feature information with the highest similarity in the current Memory Bank module is selected for fusion. The similarity calculation method and feature information fusion method are shown in Expression 3 and Expression 4 respectively.

[0057]

[0058]

[0059] Among them, o represents the local feature information of the object, k represents the feature information stored in the current Memory Bank, δ is used to ensure that the denominator is not 0, and is generally set to about 1e-5, k new Represents the new feature information after fusion, k max similarity The feature information in the current Memory Bank that is most similar to the newly acquired feature information is shown in Expression 5.

[0060]

[0061] The memory network loss takes into account that each time new feature information is learned, the current corresponding Memory Bank also generates a probability prediction of the current feature information. The calculation process is as follows Figure 4 As shown, the calculation method is shown in Expression 6.

[0062]

[0063] According to the ideal situation analysis, The value of should be 1, indicating that the current Memory Bank has fully learned the feature information. Therefore, the memory network loss designed in this embodiment of the present invention is as shown in Expression 7, which is used to measure the gap between the feature learning status of the current Memory Bank and the ideal situation. The loss is calculated and returned to the first two stages for training.

[0064]

[0065] Finally, the model is trained by continuously updating the parameters.

[0066] In this embodiment, the model parameters are trained individually for each training sample, i.e., the sample batch size is 1. The model parameters are updated using the SGD (Stochastic Gradient Descent) algorithm with an initial learning rate of 1e-3. The learning rate is then updated to 1 / 10 of the previous value after every 10 epochs. The best model data is saved to obtain the trained model.

[0067] Input a schematic diagram of the course area, and through the above three stages of image feature extraction, fusion and classification and model training, the detection results can be output to achieve target detection of the objects contained in the schematic diagram.

[0068] To explore the effectiveness of the present invention, a picture from the test data was selected for testing to explore the ability of the two solutions proposed in the present invention to handle the target detection task of the course domain diagram. The three groups of picture results obtained after adding only the information fusion module, adding the memory network module, and adding both the information fusion and memory network modules are shown as follows: Figure 5 、 6 , as shown in 7.

[0069] It can be observed that the information fusion module improves the model's ability to learn sample data features, and the memory network module improves the model's ability to learn low-frequency sample data. Figure 5 and Figure 7 As shown, the former method, which only adds a memory network module, can learn low-frequency samples, but still has difficulty distinguishing objects with similar features. The latter method, which further adds an information fusion module, can distinguish similar objects such as crabs and frogs. The two modules proposed in this embodiment of the present invention are effective for the task of detecting objects in diagrams in the curriculum field.

[0070] The experimental performance of the embodiment of the present invention under the two backbones of VGG and ResNet is as follows: Figure 8 shown.

[0071] 1) By combining global feature information with the Faster-R-CNN model, we designed a target object detection method that fuses multi-granularity visual information, namely Information fusion. We can see that the model performance improves from 2%-3% mAP to around 15% mAP, greatly enhancing the model's detection performance.

[0072] 2) It can be seen that in all cases, adding the Memory Bank module improves the model performance, and the best performance can basically reach about 19%.

[0073] 3) Combining the information fusion method with the memory enhancement method, we conducted another experimental verification on the schematic dataset and obtained the experimental results shown in the figure. We can see that the best performing model has achieved a mAP of about 22%.

[0074] Through the above ablation experiments, it can be seen that both the information fusion and memory enhancement methods proposed in the present invention have achieved relatively good performance, proving that both methods have the ability to enhance the learning ability of the model in the task of detecting low-frequency objects in schematic diagrams.

[0075] The experimental performance of the present invention on different memory slot parameters in the memory network is as follows Figure 9As shown in the figure, the number of memory slots in the memory network is a relatively important hyperparameter, representing the current memory network's capacity to retain learning results. Experimental results obtained for the number of memory slots, 3, 4, 5, and 6, were 19.5%, 20%, 22%, and 21%, respectively. A greater number of memory slots means greater memory capacity, while a lower number means less memory capacity.

[0076] Another embodiment of the present invention further provides a course domain diagram object detection system, comprising:

[0077] The feature fusion module is used to extract the global feature information of the image and the local feature information of the object to be detected and fuse them to obtain fused feature information;

[0078] A key-value pair acquisition module is used to obtain the prediction probability corresponding to the fused feature information, and to form a key-value pair with the local feature information of the object to be detected and the prediction probability;

[0079] The model training module is used to write key-value pairs into the memory network, update the key-value pairs, and train the object detection network model by transferring the loss function;

[0080] The object output module is used to input the course domain diagram into the trained object detection network model and output the objects contained in the course domain diagram.

[0081] Another embodiment of the present invention further provides an electronic device, comprising: a memory storing at least one instruction; and a processor executing the instruction stored in the memory to implement the method for detecting objects in a course domain diagram.

[0082] Another embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for detecting objects in a course domain diagram is implemented.

[0083] Exemplarily, the instructions stored in the memory may be divided into one or more modules / units, which are stored in a computer-readable storage medium and executed by the processor to implement the method for detecting objects in a course domain diagram according to the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the server.

[0084] The electronic device may be a computing device such as a smartphone, laptop, PDA, or cloud server. The electronic device may include, but is not limited to, a processor and memory. Those skilled in the art will appreciate that the electronic device may include more or fewer components, or a combination of certain components, or different components. For example, the electronic device may also include input / output devices, network access devices, buses, and the like.

[0085] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0086] The memory may be an internal storage unit of the server, such as a hard disk or memory of the server. The memory may also be an external storage device of the server, such as a plug-in hard disk equipped on the server, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Furthermore, the memory may include both an internal storage unit of the server and an external storage device. The memory is used to store the computer-readable instructions and other programs and data required by the server. The memory may also be used to temporarily store data that has been output or is about to be output.

[0087] It should be noted that the information interaction, execution process, etc. between the above-mentioned module units are based on the same concept as the method embodiment. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0088] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0089] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunications signal and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk.

[0090] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0091] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A method for detecting objects in a course domain diagram, characterized in that: include: Extract the global feature information of the image and the local feature information of the object to be detected and fuse them to obtain fused feature information; Obtain the predicted probability corresponding to the fused feature information, and form a key-value pair with the local feature information of the object to be detected and the predicted probability; Write the key-value pairs into the memory network, update the key-value pairs and train the object detection network model by transferring the loss function, input the course domain diagram to the trained object detection network model, and output the objects contained in the course domain diagram; In the step of writing the key-value pair into the memory network, the memory network is provided with a Memory Bank module based on feature information and prediction probability, and the expression of the key-value pair is , where and They are used to store feature information and prediction probability respectively. Indicates the number of object categories; In the step of forming a key-value pair of the local feature information of the object to be detected and the predicted probability, the local feature information of the object is and predicted probability The corresponding feature information and prediction probability form a key-value pair , the generated key-value pairs are passed to the Memory Bank module as the input of the memory network; In the step of updating the key-value pairs and training the object detection network model by transferring the loss function, the feature information with the highest similarity in the current Memory Bank module is selected for fusion. The calculation expressions of the similarity calculation method and the feature information fusion method are as follows: Where, Represents the local feature information of the object, Indicates the feature information stored in the current Memory Bank module. Used to ensure that the denominator is not 0, Represents the new feature information after fusion, Indicates the feature information in the current Memory Bank module that has the highest similarity with the newly acquired feature information; Each time new feature information is learned, the corresponding Memory Bank module also generates a probability prediction for the current feature information. The calculation expression is as follows: Ideally, The value of is 1, and the loss function of the memory network is as follows:

2. The method for detecting objects in a course domain diagram according to claim 1, wherein: The object detection network model adopts the Faster R-CNN network model.

3. The method for detecting objects in a course domain diagram according to claim 2, wherein: The step of extracting global feature information of the image and local feature information of the object to be detected and fusing them to obtain fused feature information comprises: Use the pre-trained Faster R-CNN network model to obtain global feature information of the image ; Use the pre-trained Faster R-CNN network model to detect objects in the course domain image and obtain Local feature information of an object ; The global feature information of the image and Local feature information of an object Fusion is performed to obtain fusion feature information .

4. The method for detecting objects in a course domain diagram according to claim 3, wherein: In the step of obtaining the prediction probability corresponding to the fused feature information, the obtained fused feature information Input the pre-trained FasterR-CNN network model to obtain the corresponding prediction probability .

5. The method for detecting objects in a course domain diagram according to claim 1, wherein: In the step of writing the key-value pair into the memory network, each time new feature information and predicted probability are obtained, it is determined whether the MemoryBank module of the current category is full. If it is not full, the memory writing phase is executed; if it is full, the memory updating phase is executed; the memory writing phase directly writes the newly obtained feature information and predicted probability key-value pair into the memory network. Add to the end of the Memory Bank module.

6. A course domain diagram object detection system, characterized in that: include: The feature fusion module is used to extract the global feature information of the image and the local feature information of the object to be detected and fuse them to obtain fused feature information; A key-value pair acquisition module is used to obtain the prediction probability corresponding to the fused feature information, and to form a key-value pair with the local feature information of the object to be detected and the prediction probability; The model training module is used to write key-value pairs into the memory network, update the key-value pairs, and train the object detection network model by transferring the loss function; The object output module is used to input the course domain diagram into the trained object detection network model and output the objects contained in the course domain diagram; In the step of writing the key-value pair into the memory network, the memory network is provided with a Memory Bank module based on feature information and prediction probability, and the expression of the key-value pair is , where and They are used to store feature information and prediction probability respectively. Indicates the number of object categories; In the step of forming a key-value pair of the local feature information of the object to be detected and the predicted probability, the local feature information of the object is and predicted probability The corresponding feature information and prediction probability form a key-value pair , the generated key-value pairs are passed to the Memory Bank module as the input of the memory network; In the step of updating the key-value pairs and training the object detection network model by transferring the loss function, the feature information with the highest similarity in the current Memory Bank module is selected for fusion. The calculation expressions of the similarity calculation method and the feature information fusion method are as follows: Where, Represents the local feature information of the object, Indicates the feature information stored in the current Memory Bank module. Used to ensure that the denominator is not 0, Represents the new feature information after fusion, Indicates the feature information in the current Memory Bank module that has the highest similarity with the newly acquired feature information; Each time new feature information is learned, the corresponding Memory Bank module also generates a probability prediction for the current feature information. The calculation expression is as follows: Ideally, The value of is 1, and the loss function of the memory network is as follows:

7. An electronic device, characterized in that: include: a memory storing at least one instruction; and A processor is configured to execute instructions stored in the memory to implement the object detection method for a course domain diagram according to any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for detecting objects in a course domain diagram according to any one of claims 1 to 5 is implemented.

Images