Time-sequence image recognition method, device and medium for archival bulk material attachment

Abstract

The present application discloses a time-series image recognition method, device and medium for the attachment of archive loose materials, which belongs to the field of artificial intelligence technology. The method comprises: using a training picture set to train a deep learning model to obtain a trained time-series image recognition model; inputting the time-series image to be identified into the time-series image recognition model to obtain a recognition result. The training picture set includes time-series position information, and the recognition result includes the category of the time-series image and the time-series position information in the category. Through the picture recognition method and device of the present invention, when the deep learning model is trained, the time-series position information is added, the position vector is calculated according to the time-series position information, and the position vector is spliced ​​into the features of the time-series image. Therefore, when the trained deep learning model recognizes the time-series image, it can not only recognize the category of the image, but also recognize the time-series position information of the image, thereby accurately identifying the time-series position of each image.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device and medium for identifying time-series images for linking archive loose materials. Background Art

[0002] In some practical scenarios, it is necessary to reorder disrupted ordered image data. For this purpose, a series of image sorting schemes have been proposed. For example, patent publication number CN106844503A proposes an image processing method comprising: receiving multiple images input by a user; determining the image type of a first image and a probability value of the first image belonging to the image type, where the first image is any one of the multiple images; and determining the order of the first image among the multiple images based on the image type of the first image and the probability value of the first image belonging to the image type. For another example, the patent application with publication number CN117275649A proposes a method and device, electronic device and storage medium for sorting document medical record images. The method includes obtaining a number of document medical record images, and extracting document header information, document footer information and document body information respectively; determining the home page image among the number of document medical record images based on the document header information and document footer information, and extracting the head text and footer text of the number of document medical record images respectively based on the document body information; combining the number of document medical record images into image pairs, and determining the adjacent confidence of the two document medical record images in the image pair based on the head text and footer text; determining the maximum probability path of the number of document medical record images based on the home page image and the adjacent confidence to sort the number of document medical record images.

[0003] With the development of artificial intelligence technology, deep learning technology has provided a new paradigm for image sorting. Compared with traditional solutions that rely on artificial rules (such as sorting based on image type probability or document structure), deep learning models achieve feature adaptive learning through data-driven. When using existing deep learning models to sort disrupted ordered image data, classification models are generally used to complete this task. The method is to use the time sequence number of each image as a classification label, and then train the classification label to improve the classification model's ability to predict the time sequence position of each image. Figure 1It's like a horse racing time series chart. These charts are sequential in time. After classification and identification, each chart should be accurately assigned a classification number (1-8), corresponding to the chronological order. For another example, in the management of loose archival materials, the contents of each person's file are chronologically ordered. Annually, due to assessments or promotions, several archival documents may be generated. These files may have multiple pages, and the specific number of pages is not fixed. Furthermore, files from different individuals may be intermixed, such as the first page of Person A's file and the third page of Person B's file. After these files are identified, the first page of File A should be linked to Person A's file, and the third page of File B should be linked to Person B's file.

[0004] Taking the existing YOLO deep learning model as an example, Figure 1 When the eight horse racing pictures shown are used for time series position prediction, the results are as follows:

[0005] Table 1: The existing YOLO model's temporal position prediction results for the horse racing image

[0006]

[0007] As can be seen from the recognition results in Table 1, due to the high similarity between the third and fourth images, and the seventh and eighth images, recognition methods using only a single image cannot effectively distinguish the temporal position relationship. For an image with a true temporal position of 3, the probabilities of the predicted temporal position being 3 and 4 are 0.48 and 0.4, respectively, showing no significant difference. For an image with a true temporal position of 4, the probabilities of the predicted temporal position being 3 and 4 are 0.35 and 0.6, respectively, showing insufficient discrimination. For an image with a true temporal position of 8, the probabilities of the predicted temporal position being 7 and 8 are 0.37 and 0.63, respectively, showing insufficient discrimination.

[0008] Existing deep learning models rely solely on the image features of a single image to determine image recognition. They fail to incorporate temporal positional correlation information into the overall image stream, resulting in inaccurate recognition of the image's temporal position. Currently, no effective solution has been proposed to this technical issue. Summary of the Invention

[0009] The embodiments of the present disclosure provide a method, device and medium for time-series image recognition for archival loose material attachment, so as to at least solve the technical problem in the prior art that the deep learning model's recognition judgment of each image is simply learned and trained based on the image features of a single image, and the time-series position correlation information is not included in the training for the entire image stream, resulting in the inability to accurately identify the time-series position of the image during recognition.

[0010] According to one aspect of an embodiment of the present disclosure, a method for recognizing sequential images is provided, comprising: training a preset deep learning model using a training image set to obtain a trained sequential image recognition model; wherein the preset deep learning model comprises a backbone network, a position fusion network, a neck structure, and a head structure; the backbone network comprises N convolutional layers connected in series, wherein the convolutional layers are used to extract features from an input image; the position fusion network comprises N fusion modules respectively connected to the outputs of the N convolutional layers of the backbone network, wherein the fusion modules are used to fuse the sequential position information of the training images in the training image set with the outputs of the convolutional layers of the backbone network; the neck structure is used to perform multi-scale feature fusion and feature enhancement on the features output by the position fusion network; the head structure is used to convert the features output by the neck structure into a recognition result; wherein N is an integer greater than or equal to 3; and inputting the sequential image to be recognized into the sequential image recognition model to obtain a recognition result; wherein the recognition result comprises the category of the sequential image and the sequential position information in the category; the sequential image is an image in which a sequential position relationship exists between different images.

[0011] According to another aspect of an embodiment of the present disclosure, a storage medium is further provided, the storage medium including a stored program, wherein when the program is run, a processor executes any one of the above methods.

[0012] According to another aspect of the embodiments of the present disclosure, a time series image recognition device is also provided, including: a training module, used to train a preset deep learning model using a training image set to obtain a trained time series image recognition model; wherein the preset deep learning model includes a backbone network, a position fusion network, a neck structure and a head structure; the backbone network includes N convolutional layers connected in series, and the convolutional layers are used to extract features from the input image; the position fusion network includes N fusion modules respectively connected to the outputs of the N convolutional layers of the backbone network, and the fusion modules are used to fuse the time series position information of the training images in the training image set with the outputs of the convolutional layers of the backbone network; the neck structure is used to perform multi-scale feature fusion and feature enhancement on the features output by the position fusion network; the head structure is used to convert the features output by the neck structure into recognition results; wherein N is an integer greater than or equal to 3; a recognition module, used to input the time series image to be recognized into the time series image recognition model to obtain a recognition result; wherein the recognition result includes the category of the time series image and the time series position information in the category; the time series image is an image with a time series position relationship between different images.

[0013] According to another aspect of the embodiment of the present disclosure, a time series image recognition device is also provided, including: a processor; and a memory, connected to the processor, for providing the processor with instructions for processing the following processing steps: using a training picture set to train a preset deep learning model to obtain a trained time series image recognition model; wherein the preset deep learning model includes a backbone network, a position fusion network, a neck structure and a head structure; the backbone network includes N convolutional layers connected in series, and the convolutional layers are used to extract features from the input image; the position fusion network includes N fusion layers respectively connected to the outputs of the N convolutional layers of the backbone network A fusion module is provided, wherein the fusion module is used to fuse the temporal position information of the training pictures in the training picture set with the output of the convolutional layer of the backbone network; the neck structure is used to perform multi-scale feature fusion and feature enhancement on the features output by the position fusion network; the head structure is used to convert the features output by the neck structure into recognition results; N is an integer greater than or equal to 3; and the temporal picture to be recognized is input into the temporal picture recognition model to obtain a recognition result; wherein the recognition result includes the category of the temporal picture and the temporal position information in the category; the temporal picture is a picture in which a temporal position relationship exists between different pictures.

[0014] The time-series image recognition method for archival bulk material attachment of this embodiment incorporates time-series position information during deep learning model training. Based on this time-series position information, a position vector is calculated and incorporated into the time-series image features. Therefore, when the trained deep learning model recognizes time-series images, it can not only identify the image category but also the image's time-series position information, thereby accurately identifying the time-series position of each image. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of this application. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation of the present disclosure. In the drawings:

[0016] Figure 1 It is a horse racing timing diagram;

[0017] Figure 2 is a hardware structure block diagram of a computing device for implementing the method according to embodiment 1 of the present disclosure;

[0018] Figure 3 1 is a flow chart of a method for recognizing sequential images according to the first aspect of Embodiment 1 of the present disclosure;

[0019] Figure 4This is one of the deep learning model structure diagrams according to the first aspect of Example 1 of the present disclosure;

[0020] Figure 5 This is the second schematic diagram of the deep learning model structure according to the first aspect of Example 1 of the present disclosure;

[0021] Figure 6 is a schematic diagram of a time-series image recognition device according to embodiment 2 of the present disclosure;

[0022] Figure 7 3 is a schematic diagram of a time-series image recognition device according to embodiment 3 of the present disclosure. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present disclosure.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] First, some nouns or terms that appear in the description of the embodiments of the present disclosure are subject to the following explanations:

[0026] BackBone: The backbone network in a deep learning model, whose main function is to extract feature representations from the input image;

[0027] Neck: The neck structure in deep learning models, whose main function is to further process and fuse the features extracted by Backbone to better meet the needs of downstream tasks (such as object detection or semantic segmentation);

[0028] Head: The head structure in the deep learning model, whose main function is to complete specific recognition tasks such as classification, regression, and segmentation based on the features output by the neck;

[0029] PIC: Picture, that is, the image input to the backbone network;

[0030] Conv: is the abbreviation of "Convolution", which means convolution operation;

[0031] C3K2: It is a structure of multiple convolutional layers and residual connections, as well as convolution kernels, which can be referred to as residual convolution kernels;

[0032] Attention: represents the attention mechanism, which simulates human visual attention and enables the model to focus more on important features by assigning different weights to different parts of the input data;

[0033] POS: In the present invention, it represents the temporal position information, and POS is the temporal position of the training image in a specific training sequence in the training image set;

[0034] PE: The abbreviation of "Positional Encoding", which means position encoding;

[0035] Concat: is the abbreviation of "Concatenation", which means concatenation operation;

[0036] Upsample: represents the upsampling operation, which is used to increase the resolution of the feature map;

[0037] Detect: represents a detection operation. In the present invention, in the target detection task, the Detect module is used to process the input feature map and predict the category and position of the target.

[0038] Example 1

[0039] According to this embodiment, an embodiment of a time-series image recognition method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0040] The method embodiment provided in this embodiment can be executed in a mobile terminal, a computer terminal, a server or a similar computing device. Figure 2 The hardware structure block diagram of a computing device for realizing time series image recognition is shown in FIG. Figure 2As shown, a computing device may include one or more processors (the processor may include, but is not limited to, a microprocessor (MCU) or a programmable logic device (FPGA) or other processing device), a memory for storing data, a transmission device for communication functions, and an input / output interface. The memory, transmission device, and input / output interface are connected to the processor via a bus. In addition, it may also include: a display, a keyboard, and a cursor control device connected to the input / output interface. Those skilled in the art will understand that Figure 2 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 2 More or fewer components than shown, or with Figure 2 Different configurations shown.

[0041] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry." The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuitry may be a single, independent processing module, or may be fully or partially integrated into any of the other components of the computing device. As discussed in the embodiments of the present disclosure, the data processing circuitry functions as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0042] The memory can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the timing image in the embodiment of the present disclosure. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the timing image recognition method of the above-mentioned application. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely located relative to the processor, and these remote memories may be connected to the computing device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0043] The transmission device is used to receive or send data via a network. Specific examples of the aforementioned network may include a wireless network provided by a communications provider of the computing device. In one embodiment, the transmission device includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0044] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computing device.

[0045] It should be noted that, in some optional embodiments, the above Figure 2 The computing device shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. Figure 2 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computing device described above.

[0046] Under the above operating environment, according to the first aspect of this embodiment, a time sequence picture recognition method is provided. Figure 2 The temporal image recognition system or device shown in is implemented. Figure 3 A schematic diagram of the process is shown in FIG. Figure 3 As shown, the method includes:

[0047] S302: Use the training picture set to train the preset deep learning model to obtain a trained temporal picture recognition model; wherein the preset deep learning model includes a backbone network, a position fusion network, a neck structure and a head structure; the backbone network includes N convolutional layers connected in series, and the convolutional layers are used to extract features from the input image; the position fusion network includes N fusion modules respectively connected to the outputs of the N convolutional layers of the backbone network, and the fusion modules are used to fuse the temporal position information of the training pictures in the training picture set with the outputs of the convolutional layers of the backbone network; the neck structure is used to perform multi-scale feature fusion and feature enhancement on the features output by the position fusion network; the head structure is used to convert the features output by the neck structure into recognition results; N is an integer greater than or equal to 3;

[0048] S304: Input the time-series images to be identified into the time-series image recognition model to obtain a recognition result, wherein the recognition result includes the category of the time-series images and the time-series position information within the category; the time-series images are images with a time-series position relationship between different images.

[0049] In the present invention, a time sequence picture is a picture with a time sequence relationship between different pictures, and the time sequence position is the order of the pictures in the time sequence, that is, from the time sequence, a certain picture should be in which picture in the picture set. It can also be understood as, from the time sequence, which picture should be the previous and next picture of a specific picture. For example Figure 1As shown in the figure, there is a temporal relationship between the first to eighth horse racing images, which corresponds to the temporal sequence of a horse's movements. This means that when the deep learning model calculates the next image after the first horse racing image, the probability of image 2 being the next image is significantly higher than that of the other images.

[0050] For another example, in the linked management of scattered archival materials, each page is treated as an image, with a time and reason for its creation. The content of each person's archive follows a chronological order. For example, new archive documents may be generated due to performance reviews or promotions, or due to job changes or organizational transfers. These archive documents may have multiple pages, and the specific number of pages is not fixed. Furthermore, when linking multiple archives, archives from different individuals may be intermixed, such as the first page of Person A's archive A and the third page of Person B's archive B. After these archives are identified, the first page of Person A's archive should be linked to Person A's archive A, and the third page of Person B's archive should be linked to Person B's archive B. If Person A's archive A consists of five pages, then pages 1 through 5 of Person A's archive A also follow a sequential order. For example, if a five-page archive describes an organizational change, each page contains detailed information about the change. These five pages have a temporal relationship with each other in describing the organizational change, and these relationships conform to the sequential order of the document describing the organizational change. Similarly, if Person B's file contains File B and File C, with File B representing the transition from elementary school to junior high school and File C representing the transition from high school to university, then File C should be placed after File B in terms of temporal position. If Person B's File C has three pages, recording the transition from high school to university, then these three pages also have a temporal positional relationship in describing the transition from high school to university. This is similar to the temporal relationship between the beginning and end of an article; the end cannot be placed before the beginning.

[0051] Optionally, the training image set is a collection of pre-labeled images, and each image is annotated with temporal position information.

[0052] In the present invention, based on the existing deep learning model architecture, a position fusion network is added to fuse the temporal position information of the training images in the training image set with the output of the convolutional layer of the backbone network, thereby adding the temporal position information to the training process of the model, so that the trained deep learning model can accurately predict the temporal position of the temporal image. In the prior art, the YOLO deep learning model includes a backbone network Backbone, a neck structure Neck and a head structure Head. The method of the present invention adds a position fusion network Posc, such as Figure 4As shown in FIG. , the backbone network 402 outputs image feature information, the position fusion network 404 fuses the image feature information output by the backbone network 402 with the position information of the time-series image, and inputs the fused feature information into the neck structure 406 for multi-scale feature fusion and feature enhancement. Finally, the head structure 408 converts the features output by the neck structure 406 into a recognition result output. The recognition result includes the category of the time-series image and the time-series position information within the category.

[0053] As an optional example, the deep learning model of the present invention is as follows Figure 5 As shown, specifically including:

[0054] 1. Backbone network:

[0055] The backbone network extracts low-level to high-level feature representations from the input image PIC, and includes N serially connected convolutional layers. The structures of the first convolutional layer to the N-1th convolutional layer include a convolution operation module Conv and a residual convolution kernel C3K2. The Nth convolutional layer includes a convolution operation module Conv, a residual convolution kernel C3K2, and an attention mechanism module Attention. Figure 5 In the example, N is 3. The backbone network consists of the first, second, and third convolutional layers. The first convolutional layer includes a ConvNet and a C3K2 layer, the second convolutional layer includes a ConvNet and a C3K2 layer, and the third convolutional layer includes a ConvNet, a C3K2 layer, and an Attention layer. The output of the first convolutional layer is passed to the second convolutional layer for further processing, and the output of the second convolutional layer is passed to the third convolutional layer for processing. Each convolutional layer also outputs image features, which are passed to the position fusion network (Posc) for further fusion processing.

[0056] Thus, Backbone gradually extracts local and global features of the image through multiple layers of convolution and pooling operations. The output of each layer can be regarded as the feature representation of the input image PIC at different scales and semantic levels.

[0057] 2. Position Fusion Network Posc:

[0058] Posc includes N serially connected fusion layers, each of which includes a position encoding module PE and a concatenation module Concat. During training, the temporal position information POS is input into the N fusion layers for fusion processing. The connection relationship when N is equal to 3 is as follows Figure 5 shown.

[0059] The position encoding module (PE) encodes temporal position information, which helps the model understand the temporal order of elements in a sequence. The concatenation module (Concat) concatenates the image feature tensors output by the backbone network along a specific dimension to generate a new feature tensor, which is then fed into the neck network (Neck) for further processing.

[0060] 3. Neck structure:

[0061] The neck structure is used to perform multi-scale feature fusion and feature enhancement on the features output by the position fusion network, including N-1 serially connected upward processing modules and serially connected downward processing modules. The upward processing module includes a serially connected upsampling module Upsample, a splicing module Concat and a residual convolution kernel C3K2, and the downward processing module includes a serially connected convolution operation module Conv, a splicing module Concat and a residual convolution kernel C3K2.

[0062] like Figure 5 As shown in the figure, when N is 3, the processing relationship is as follows: the output of the third layer of the position fusion network is passed to both the first upward processing module and the second downward processing module. The output of the second layer of the position fusion network is passed to the concatenation module Concat of the first upward processing module. The output of the first layer of the position fusion network is passed to the concatenation module Concat of the second upward processing module.

[0063] After the residual convolution kernel C3K2 of the first upward processing module is processed, the result is output to the up-sample module Upsample of the second upward processing module and the concatenation module Concat of the first downward processing module at the same time. After the residual convolution kernel C3K2 of the second upward processing module is processed, the result is output to the convolution operation module Conv of the first downward processing module and the first detection module Detect of the head structure at the same time. After the residual convolution kernel C3K2 of the first downward processing module is processed, the result is output to the convolution operation module Conv of the second downward processing module and the second detection module Detect of the head structure at the same time. After the residual convolution kernel C3K2 of the first downward processing module is processed, the result is output to the third detection module Detect of the head structure.

[0064] 4. Head structure:

[0065] The head structure is used to convert the features output by the neck structure into recognition results. The head structure includes N detection modules Detect. Figure 5 In the figure, when N is equal to 3, the three detection modules of the head structure are connected to the neck module Neck respectively.

[0066] As an optional example, the preset deep learning model is trained using the training picture set, using Figure 5 The deep learning model shown in the figure is trained, including:

[0067] Step 1: Determine the temporal position information of the training pictures in the training picture set;

[0068] Step 2: Input the training image into the backbone network, input the temporal position information into the position fusion network, and fuse the position information with the features of the corresponding layer of the backbone network through the N fusion modules;

[0069] Step 3: Perform multi-scale feature fusion and feature enhancement on the features output by the position fusion network through the neck structure;

[0070] Step 4: Output the prediction result through the header structure, where the prediction result includes the probability distribution of the category to which the picture belongs and the temporal position information of the picture;

[0071] Step 5: Calculate the category prediction loss and the time series position prediction loss, perform weighted summation of the category prediction loss and the time series position prediction loss to obtain the total loss, and update the parameters of the deep learning model according to the total loss until the total loss converges;

[0072] Among them, the parameters of the deep learning model include one or a combination of the following: classification and regression parameters of the head structure, feature fusion weight parameters of the neck structure, temporal position information fusion parameters of the position fusion network, and convolutional layer weight parameters of the backbone network.

[0073] It should be noted that the parameters of the deep learning model may vary depending on the number of layers and structure of the deep learning model, and are determined based on specific needs.

[0074] As an optional example, the fusing of the temporal position information of the training pictures in the training picture set with the output of the convolutional layer of the backbone network includes: position encoding the temporal position information to generate a position vector; and splicing the position vector with the output of the convolutional layer of the backbone network.

[0075] In the present invention, temporal position information refers to the specific position of each training image in the training image set within its category according to the temporal relationship. For example, suppose there are 10 horse racing pictures, and each horse has 8 horse racing pictures (such as Figure 1 As shown in the figure, a total of 80 horse racing pictures constitute the training picture set, which is divided into 10 categories, corresponding to 10 horses respectively. In each category, the 8 pictures have a fixed order in terms of temporal relationship, and each picture has a temporal position for that. For example Figure 1 As shown, the timing position information is 1 to 8 respectively.

[0076] As an optional example, performing position encoding on the time series position information to generate a position vector includes: for each fusion module, calculating the position vector according to the following formula:

[0077] ;

[0078] in, POS is the temporal position of the training picture in a specific training sequence in the training picture set, dim is the dimension of the position vector, i is the dimension depth index of the position vector; the position vector The timing position is POS The training images are indexed in the dimension depth i The position vector on .

[0079] As an optional example, the dim It is determined according to the dimension depth of the convolutional layer of the backbone network connected to the fusion module.

[0080] As an optional example, the value range of the dimension depth index i is based on dim For example, when dim When it is 256, the value range of i is 0~255; when dim When it is 512, the value range of i is 0~511; when dim When it is 1024, the value range of i is 0~1023, and so on.

[0081] When will Figure 1 When the 8 pictures with temporal position relationship shown are used as training samples, POS The values ​​of are 0 to 7. Assume that Figure 5 The three outputs in Backbone correspond to dimensions of 80×80×256, 40×40×512, and 20×20×1024. Therefore, we need to generate corresponding 256-, 512-, and 1024-dimensional position vectors for each image. The generator matrices for the 256-dimensional position vectors corresponding to these eight images are shown in Table 2. The generator matrices for the 512- and 1024-dimensional position vectors are similar.

[0082] Table 2

[0083]

[0084] In Table 2 above, Dim 0 to Dim 255 stored in the first row correspond to the component index of the 256-dimensional position vector, and Pos 0 to Pos 7 stored in the first column represent the time sequence position 0 to 7 (i.e. pos= 0~7), each row and column intersection cell records the calculation formula of the position vector component (Dim, Pos) corresponding to the picture at that time sequence position. Figure 1 Take the first picture in the figure as an example, we can use the formula Calculate the value of the Dim 0 component in the 256-dimensional position vector using the formula Calculate the value of the Dim 1 component in the 256-dimensional position vector. Continue this process to obtain all component values ​​of the 256-dimensional position vector corresponding to the image at temporal position 0. Finally, aggregate all component values ​​to generate the 256-dimensional position vector corresponding to the image at temporal position 0.

[0085] Through the time series image recognition method of the present invention, in the training stage of the deep learning model, the time series position information is used as one of the training input information and participates in the entire training process, so that the trained deep learning model can significantly improve its prediction ability of time series position. Figure 1 The results of recognizing the horse racing picture shown using the existing deep learning model are shown in Table 1 above, while the results of recognizing the time-series picture using the time-series picture recognition method of the present invention are shown in Table 3 below.

[0086] Table 3: Time sequence position prediction results of the horse racing image by the time sequence image recognition method of the present invention

[0087]

[0088] As shown in Table 3, although the similarity between the 3rd and 4th pictures, and the 7th and 8th pictures is relatively high, the accuracy of the recognition results of the temporal position is significantly improved after being recognized by the method of the present invention. For pictures with a true temporal position of 3, the probability that the temporal position prediction result is 3 increases from 0.48 to 0.82, while the probability that the temporal position prediction result is 4 decreases from 0.4 to 0.18, and the prediction accuracy is significantly improved. For pictures with a true temporal position of 4, the probability that the temporal position prediction result is 3 decreases from 0.35 to 0.18, while the probability that the temporal position prediction result is 4 increases from 0.6 to 0.81, and the prediction accuracy is significantly improved. For pictures with a true temporal position of 8, the probability that the temporal position prediction result is 7 decreases from 0.37 to 0.1, while the probability that the temporal position prediction result is 8 increases from 0.63 to 0.9, and the prediction accuracy is also significantly improved.

[0089] The time-series image recognition method of this embodiment incorporates time-series position information during deep learning model training. Based on this information, a position vector is calculated and incorporated into the features of the time-series image. Therefore, when the trained deep learning model recognizes time-series images, it can not only identify the image category but also the image's time-series position information, thereby accurately identifying the time-series position of each image.

[0090] Furthermore, according to a second aspect of this embodiment, a storage medium is provided, wherein the storage medium includes a stored program, wherein when the program is run, a processor executes any one of the above methods.

[0091] Therefore, according to this embodiment, in the training stage of the deep learning model, the time series position information is used as one of the input information for training and participates in the entire training process, so that the trained deep learning model can significantly improve its prediction ability in time series position.

[0092] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that the present invention is not limited by the order of the actions described, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0093] Through the description of the above embodiments, those skilled in the art will clearly understand that the methods according to the above embodiments can be implemented using software plus the necessary general-purpose hardware platform. Of course, hardware can also be used, but in many cases the former is a more preferred embodiment. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, or optical disk) and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in various embodiments of the present invention.

[0094] Example 2

[0095] Figure 6 The device for recognizing a time sequence image according to this embodiment is shown, which corresponds to the method according to the first aspect of embodiment 1. Figure 6As shown, the device includes: a training module 610, which is used to train a preset deep learning model using a training picture set to obtain a trained time series picture recognition model; wherein the preset deep learning model includes a backbone network, a position fusion network, a neck structure and a head structure; the backbone network includes N convolutional layers connected in series, and the convolutional layers are used to extract features from the input image; the position fusion network includes N fusion modules respectively connected to the outputs of the N convolutional layers of the backbone network, and the fusion modules are used to fuse the time series position information of the training pictures in the training picture set with the output of the convolutional layer of the backbone network; the neck structure is used to perform multi-scale feature fusion and feature enhancement on the features output by the position fusion network; the head structure is used to convert the features output by the neck structure into recognition results; N is an integer greater than or equal to 3; a recognition module 620, which is used to input the time series picture to be recognized into the time series picture recognition model to obtain a recognition result; wherein the recognition result includes the category of the time series picture and the time series position information in the category; the time series picture is a picture with a time series position relationship between different pictures.

[0096] Optionally, the use of a training picture set to train a preset deep learning model includes: determining the temporal position information of the training pictures in the training picture set; inputting the training pictures into the backbone network, inputting the temporal position information into the position fusion network, and fusing the position information with the features of the corresponding layer of the backbone network through the N fusion modules; performing multi-scale feature fusion and feature enhancement on the features output by the position fusion network through the neck structure; outputting a prediction result through the head structure, wherein the prediction result includes the probability distribution of the category to which the picture belongs and the temporal position information of the picture; calculating the category prediction loss and the temporal position prediction loss, weightedly summing the category prediction loss and the temporal position prediction loss to obtain a total loss, and updating the parameters of the deep learning model according to the total loss until the total loss converges; wherein the parameters of the deep learning model include one or a combination of the following: classification and regression parameters of the head structure, feature fusion weight parameters of the neck structure, temporal position information fusion parameters of the position fusion network, and convolutional layer weight parameters of the backbone network.

[0097] Optionally, the fusing of the temporal position information of the training pictures in the training picture set with the output of the convolutional layer of the backbone network includes: position encoding the temporal position information to generate a position vector; and splicing the position vector with the output of the convolutional layer of the backbone network.

[0098] Optionally, performing position encoding on the time series position information to generate a position vector includes: for each fusion module, calculating the position vector according to the following formula: ;in, POS is the temporal position of the training picture in a specific training sequence in the training picture set, dim is the dimension of the position vector, i is the dimension depth index of the position vector; the position vector The timing position is POS The training images are indexed in the dimension depth i The position vector on .

[0099] Optionally, the dim It is determined according to the dimension depth of the convolutional layer of the backbone network connected to the fusion module.

[0100] Optionally, the value range of the dimension depth index i is based on dim to determine the size.

[0101] According to this embodiment, during the training phase of the deep learning model, the temporal position information is used as one of the input information for training and participates in the entire training process, thereby significantly improving the prediction ability of the deep learning model after training in terms of temporal position.

[0102] Example 3

[0103] Figure 7 The device for recognizing a time sequence image according to this embodiment is shown, which corresponds to the method according to the first aspect of embodiment 1. Figure 7 As shown, the device includes: a processor 710; and a memory 720, which is connected to the processor 710 and is used to provide the processor 710 with instructions for processing the following processing steps: using a training picture set to train a preset deep learning model to obtain a trained temporal picture recognition model; wherein the preset deep learning model includes a backbone network, a position fusion network, a neck structure and a head structure; the backbone network includes N convolutional layers connected in series, and the convolutional layers are used to extract features from the input image; the position fusion network includes N fusion modules respectively connected to the outputs of the N convolutional layers of the backbone network, The fusion module is used to fuse the temporal position information of the training pictures in the training picture set with the output of the convolutional layer of the backbone network; the neck structure is used to perform multi-scale feature fusion and feature enhancement on the features output by the position fusion network; the head structure is used to convert the features output by the neck structure into recognition results; N is an integer greater than or equal to 3; the temporal picture to be recognized is input into the temporal picture recognition model to obtain a recognition result; wherein, the recognition result includes the category of the temporal picture and the temporal position information in the category; the temporal picture is a picture in which there is a temporal position relationship between different pictures.

[0104] Optionally, the use of a training picture set to train a preset deep learning model includes: determining the temporal position information of the training pictures in the training picture set; inputting the training pictures into the backbone network, inputting the temporal position information into the position fusion network, and fusing the position information with the features of the corresponding layer of the backbone network through the N fusion modules; performing multi-scale feature fusion and feature enhancement on the features output by the position fusion network through the neck structure; outputting a prediction result through the head structure, wherein the prediction result includes the probability distribution of the category to which the picture belongs and the temporal position information of the picture; calculating the category prediction loss and the temporal position prediction loss, weightedly summing the category prediction loss and the temporal position prediction loss to obtain a total loss, and updating the parameters of the deep learning model according to the total loss until the total loss converges; wherein the parameters of the deep learning model include one or a combination of the following: classification and regression parameters of the head structure, feature fusion weight parameters of the neck structure, temporal position information fusion parameters of the position fusion network, and convolutional layer weight parameters of the backbone network.

[0105] Optionally, the fusing of the temporal position information of the training pictures in the training picture set with the output of the convolutional layer of the backbone network includes: position encoding the temporal position information to generate a position vector; and splicing the position vector with the output of the convolutional layer of the backbone network.

[0106] Optionally, performing position encoding on the time series position information to generate a position vector includes: for each fusion module, calculating the position vector according to the following formula: ;in, POS is the temporal position of the training picture in a specific training sequence in the training picture set, dim is the dimension of the position vector, i is the dimension depth index of the position vector; the position vector The timing position is POS The training images are indexed in the dimension depth i The position vector on .

[0107] Optionally, the dim It is determined according to the dimension depth of the convolutional layer of the backbone network connected to the fusion module.

[0108] Optionally, the value range of the dimension depth index i is based on dim to determine the size.

[0109] According to this embodiment, during the training phase of the deep learning model, the temporal position information is used as one of the input information for training and participates in the entire training process, thereby significantly improving the prediction ability of the deep learning model after training in terms of temporal position.

[0110] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0111] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0112] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0113] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0114] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0115] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), a mobile hard drive, a magnetic disk, or an optical disk.

[0116] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A time series image recognition method, characterized in that: include: A preset deep learning model is trained using a training picture set to obtain a trained temporal picture recognition model; wherein the preset deep learning model includes a backbone network, a position fusion network, a neck structure, and a head structure; the backbone network includes N convolutional layers connected in series, and the convolutional layers are used to extract features from the input image; the position fusion network includes N fusion modules respectively connected to the outputs of the N convolutional layers of the backbone network, and the fusion modules are used to fuse the temporal position information of the training pictures in the training picture set with the outputs of the convolutional layers of the backbone network; the neck structure is used to perform multi-scale feature fusion and feature enhancement on the features output by the position fusion network; the head structure is used to convert the features output by the neck structure into recognition results; and N is an integer greater than or equal to 3; Inputting the time-series images to be identified into the time-series image recognition model to obtain a recognition result; wherein the recognition result includes the category of the time-series images and the time-series position information within the category; the time-series images are images in which a time-series position relationship exists between different images; The fusing the temporal position information of the training pictures in the training picture set with the output of the convolutional layer of the backbone network includes: Position-encoding the time series position information to generate a position vector; Concatenate the position vector with the output of the convolutional layer of the backbone network; The performing position encoding on the time series position information to generate a position vector includes: For each fusion module, the position vector is calculated according to the following formula: ； Wherein, pos is the temporal position of the training picture in the specific training sequence of the training picture set, dim is the dimension size of the position vector, and i is the dimension depth index of the position vector; Position vector It is the position vector of the training image with the temporal position pos at the dimension depth index i.

2. The method according to claim 1, characterized in that The method of using the training image set to train the preset deep learning model includes: Determining temporal position information of the training pictures in the training picture set; Inputting the training image into the backbone network, inputting the temporal position information into the position fusion network, and fusing the position information with the features of the corresponding layer of the backbone network through the N fusion modules; Performing multi-scale feature fusion and feature enhancement on the features output by the position fusion network through the neck structure; Outputting a prediction result through the header structure, wherein the prediction result includes a probability distribution of the category to which the picture belongs and temporal position information of the picture; Calculating the category prediction loss and the temporal position prediction loss, performing weighted summation of the category prediction loss and the temporal position prediction loss to obtain a total loss, and updating the parameters of the deep learning model according to the total loss until the total loss converges; Among them, the parameters of the deep learning model include one or a combination of the following: classification and regression parameters of the head structure, feature fusion weight parameters of the neck structure, temporal position information fusion parameters of the position fusion network, and convolutional layer weight parameters of the backbone network.

3. The method according to claim 1, characterized in that The dim is determined according to the dimensional depth of the convolutional layer of the backbone network connected to the fusion module.

4. The method according to claim 1, wherein The value range of the dimension depth index i is determined according to the size of dim.

5. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is run, the processor executes the method according to any one of claims 1 to 4.

6. A time-series image recognition device, characterized in that: include: A training module, configured to train a preset deep learning model using a training picture set to obtain a trained temporal picture recognition model; wherein the preset deep learning model comprises a backbone network, a position fusion network, a neck structure, and a head structure; the backbone network comprises N convolutional layers connected in series, the convolutional layers being used to extract features from input images; the position fusion network comprises N fusion modules respectively connected to the outputs of the N convolutional layers of the backbone network, the fusion modules being used to fuse the temporal position information of the training pictures in the training picture set with the outputs of the convolutional layers of the backbone network; the neck structure being used to perform multi-scale feature fusion and feature enhancement on the features output by the position fusion network; the head structure being used to convert the features output by the neck structure into recognition results; and N is an integer greater than or equal to 3; a recognition module, configured to input the time-series images to be recognized into the time-series image recognition model to obtain a recognition result; wherein the recognition result includes the category of the time-series image and the time-series position information within the category; the time-series images are images in which a time-series position relationship exists between different images; The fusing the temporal position information of the training pictures in the training picture set with the output of the convolutional layer of the backbone network includes: Position-encoding the time series position information to generate a position vector; Concatenate the position vector with the output of the convolutional layer of the backbone network; The performing position encoding on the time series position information to generate a position vector includes: For each fusion module, the position vector is calculated according to the following formula: ； Wherein, pos is the temporal position of the training picture in the specific training sequence of the training picture set, dim is the dimension size of the position vector, and i is the dimension depth index of the position vector; Position vector It is the position vector of the training image with the temporal position pos at the dimension depth index i.

7. The device according to claim 6, characterized in that The method of using the training image set to train the preset deep learning model includes: Determining temporal position information of the training pictures in the training picture set; Inputting the training image into the backbone network, inputting the temporal position information into the position fusion network, and fusing the position information with the features of the corresponding layer of the backbone network through the N fusion modules; Performing multi-scale feature fusion and feature enhancement on the features output by the position fusion network through the neck structure; Outputting a prediction result through the header structure, wherein the prediction result includes a probability distribution of the category to which the picture belongs and temporal position information of the picture; Calculating the category prediction loss and the temporal position prediction loss, performing weighted summation of the category prediction loss and the temporal position prediction loss to obtain a total loss, and updating the parameters of the deep learning model according to the total loss until the total loss converges; Among them, the parameters of the deep learning model include one or a combination of the following: classification and regression parameters of the head structure, feature fusion weight parameters of the neck structure, temporal position information fusion parameters of the position fusion network, and convolutional layer weight parameters of the backbone network.

8. A time-series image recognition device, characterized in that: include: processor; as well as A memory, connected to the processor, configured to provide the processor with instructions for processing the following processing steps: A preset deep learning model is trained using a training picture set to obtain a trained temporal picture recognition model; wherein the preset deep learning model includes a backbone network, a position fusion network, a neck structure, and a head structure; the backbone network includes N convolutional layers connected in series, and the convolutional layers are used to extract features from the input image; the position fusion network includes N fusion modules respectively connected to the outputs of the N convolutional layers of the backbone network, and the fusion modules are used to fuse the temporal position information of the training pictures in the training picture set with the outputs of the convolutional layers of the backbone network; the neck structure is used to perform multi-scale feature fusion and feature enhancement on the features output by the position fusion network; the head structure is used to convert the features output by the neck structure into recognition results; and N is an integer greater than or equal to 3; Inputting the time-series images to be identified into the time-series image recognition model to obtain a recognition result; wherein the recognition result includes the category of the time-series images and the time-series position information within the category; the time-series images are images in which a time-series position relationship exists between different images; The fusing the temporal position information of the training pictures in the training picture set with the output of the convolutional layer of the backbone network includes: Position-encoding the time series position information to generate a position vector; Concatenate the position vector with the output of the convolutional layer of the backbone network; The performing position encoding on the time series position information to generate a position vector includes: For each fusion module, the position vector is calculated according to the following formula: ； Wherein, pos is the temporal position of the training picture in the specific training sequence of the training picture set, dim is the dimension size of the position vector, and i is the dimension depth index of the position vector; Position vector It is the position vector of the training image with the temporal position pos at the dimension depth index i.

Images