Feature extraction model training method and apparatus, and time series data processing method and apparatus

Abstract

The present application relates to the technical field of data processing, and discloses a feature extraction model training method and apparatus, and a time series data processing method and apparatus. The feature extraction model training method comprises: acquiring a plurality of pieces of time series data from a plurality of task scenarios; acquiring a plurality of sample pairs on the basis of the plurality of pieces of time series data, the plurality of sample pairs including a positive sample pair and a negative sample pair, the positive sample pair comprising first time series data and first image data, the negative sample pair comprising the first time series data and second image data, the first image data being image data obtained by performing modal conversion on the first time series data, the second image data being image data obtained by performing modal conversion on second time series data, and the second time series data being different from the first time series data; and training a time series feature extraction model on the basis of the positive sample pair and the negative sample pair, so as to obtain a trained time series feature extraction model. By means of the method, trained models can learn not only sequence representations of time series modalities but also pattern representations of image modalities, thereby improving model feature extraction capabilities.

Description

Training method of feature extraction model, processing method and device of time series data

[0001] The present application claims priority from the Chinese patent application No. 202411291978.8, filed on September 13, 2024, and entitled "Training method of feature extraction model, processing method and device of time series data", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the technical field of data processing, in particular to a training method of a feature extraction model, a processing method and device of time series data. BACKGROUND

[0003] In the field of data processing, time series processing has a wide range of applications in various fields, such as power generation prediction in the energy field, weather prediction in the weather field, electrocardiogram diagnosis or motion posture analysis in the medical field, etc. In the above-mentioned various time series data processing task scenarios, a feature extraction model needs to be trained through time series data samples, and then the actual time series data processing is realized through the trained feature extraction model.

[0004] In the related art, a modeling method of case by case is usually used to train the model, that is, for each task scenario, the model is trained according to a large amount of time series data of the current task scenario to obtain a feature extraction model strongly related to the current business scenario.

[0005] However, for the task scenarios with less samples or annotations in the time series field, the amount of time series data available for model training is also small, which is difficult to meet the model training requirements in the related art. Moreover, since the feature extraction model obtained in the related art is strongly related to the current business scenario, the generalization performance of the feature extraction model is poor, and it is difficult to solve the cold start problem of the above-mentioned task scenarios with less samples or annotations. SUMMARY

[0006] The present application provides a training method of a feature extraction model, a processing method and device of time series data, which are used to solve the problems in the related art.

[0007] In a first aspect, a training method of a feature extraction model is provided. The method comprises: obtaining a plurality of time series data, the plurality of time series data being derived from a plurality of task scenarios; obtaining a plurality of sample pairs based on the plurality of time series data, the plurality of sample pairs comprising positive sample pairs and negative sample pairs, the positive sample pairs comprising first time series data and first image data, and the negative sample pairs comprising the first time series data and second image data; the first time series data being any time series data in the plurality of time series data, the first image data being image data obtained by modal conversion of the first time series data, and the second image data being image data obtained by modal conversion of second time series data, the second time series data being time series data other than the first time series data in the plurality of time series data; and training a time series feature extraction model according to the positive sample pairs and the negative sample pairs to obtain a trained time series feature extraction model, the time series feature extraction model being used to extract a feature vector of time series data.

[0008] In the method, the time series feature extraction model is trained by using time series data of multiple task scenarios, so that the trained time series feature extraction model can be applied to various business scenarios and has good generalization performance. In the training process, unsupervised contrast learning is used to avoid dependence on labels. Furthermore, the time series feature extraction model is trained by using positive and negative sample pairs of time series data and image data, so that the trained time series feature extraction model can learn both sequential representations such as periodicity and trend of time series modal data and pattern representations such as structure and shape of image modal data, thereby improving the feature extraction capability of the trained time series feature extraction model and solving the cold start problem of a task scenario with less samples or labels.

[0009] In a possible implementation, the positive sample pairs further comprise first time series enhancement data and second time series enhancement data, and the negative sample pairs further comprise the first time series enhancement data and third time series enhancement data; the first time series enhancement data is time series data obtained by enhancing first sub-time series data by using a first enhancement manner, the second time series enhancement data is time series data obtained by enhancing second sub-time series data by using the first enhancement manner, and the third time series enhancement data is time series data obtained by enhancing the first sub-time series data or the second sub-time series data by using a second enhancement manner; the first sub-time series data and the second sub-time series data are two different sub-time series data in the first time series data, and the first time series data is any time series data in the plurality of time series data.

[0010] In the manner, the time series feature extraction model is further trained by using positive and negative sample pairs of time series enhancement data in multiple enhancement manners under multiple perspectives. Since the diversity of the time series data is increased by using the enhancement manners, the generalization capability of the trained time series feature is stronger.

[0011] In a possible implementation, the method further includes: training the visual feature extraction model according to the positive sample pair and the negative sample pair to obtain a trained visual feature extraction model, and the visual feature extraction model is configured to extract a feature vector of the image data. In this way, the visual feature extraction model is trained together with the time sequence feature extraction model through the positive sample pair and the negative sample pair, so that the feature extraction capability of the visual feature extraction model is improved.

[0012] In a possible implementation, the visual feature extraction model is a model obtained by training a visual base model through a plurality of image data, and the visual feature extraction model has the capability of extracting a structural information representation in the image data. In this way, the trained model is directly used in the visual feature extraction model, so that the time sequence feature extraction model can learn the capability of extracting the structural information representation of the visual feature extraction model in the contrast learning process.

[0013] In a second aspect, a method for processing time sequence data is provided, which includes: obtaining time sequence data to be processed; extracting a feature vector of the time sequence data based on a feature extraction model, the feature extraction model including a time sequence feature extraction model and / or a visual feature extraction model, the time sequence feature extraction model being a time sequence feature extraction model trained by the training method in the first aspect, and the visual feature extraction model being a visual feature extraction model trained by the training method in the first aspect; and obtaining a processing result of the time sequence data based on the feature vector of the time sequence data.

[0014] In the method, the feature extraction model trained by the training method in the first aspect is used to extract the feature vector of the time sequence data to be processed. Since the training method in the first aspect has high feature extraction capability, the extracted feature vector of the time sequence data to be processed is more accurate and can better represent the features of the time sequence data to be processed, so that the processing result obtained based on the feature vector is more accurate.

[0015] In a possible implementation, the time sequence data to be processed belongs to a first task scenario, and the extracting of the feature vector of the time sequence data based on the feature extraction model includes: fine-tuning the feature extraction model through a plurality of first time sequence data in the first task scenario to obtain a fine-tuned feature extraction model; and calling the fine-tuned feature extraction model to obtain the feature vector of the time sequence data. In this way, the feature extraction model is fine-tuned through the plurality of first time sequence data in the first task scenario, so that the fine-tuned feature extraction model can learn the knowledge in the first task scenario, and the feature extraction capability of the model in the first task scenario can be further improved.

[0016] In a possible implementation, the feature extraction model is fine-tuned through the plurality of first time sequence data in the first task scenario, to obtain a fine-tuned feature extraction model, including: obtaining a plurality of first sample pairs based on the plurality of first time sequence data, the plurality of first sample pairs including positive first sample pairs and negative first sample pairs; fine-tuning the feature extraction model according to the positive first sample pairs and the negative first sample pairs to obtain the fine-tuned feature extraction model. The feature extraction model is fine-tuned in a contrast learning manner, so that the fine-tuned feature extraction model can better learn the knowledge in the first task scenario. Optionally, the fine-tuning manner can refer to the training manner in the first aspect.

[0017] In a possible implementation, the feature vector of the time sequence data is extracted based on the feature extraction model, including: calling the feature extraction model to obtain the feature vector of the time sequence data. In this manner, the pre-trained feature extraction model is directly used to extract the feature vector of the time sequence data, without the need for a fine-tuning process, thereby improving the processing efficiency.

[0018] In a possible implementation, the processing result of the time sequence data is obtained according to the feature vector of the time sequence data, including: obtaining the processing result of the time sequence data according to the feature vector of the time sequence data and a task network, the task network including a classification network, a prediction network, or a regression network. Different task networks are used to implement the mapping from the feature vector to the processing result in different task scenarios, so that the processing result is more accurate.

[0019] In a third aspect, a feature extraction model training apparatus is provided, configured to perform the method in the first aspect or any possible implementation of the first aspect. Specifically, the feature extraction model training apparatus includes modules for performing the method in the first aspect or any possible implementation of the first aspect.

[0020] In a possible implementation, the apparatus includes: an obtaining module, configured to obtain a plurality of time sequence data, the plurality of time sequence data being derived from a plurality of task scenarios; the obtaining module is further configured to obtain a plurality of sample pairs based on the plurality of time sequence data, the plurality of sample pairs including positive sample pairs and negative sample pairs, the positive sample pairs including first time sequence data and first image data, the negative sample pairs including the first time sequence data and second image data; the first time sequence data is any time sequence data in the plurality of time sequence data, the first image data is image data obtained by modal conversion on the first time sequence data, the second image data is image data obtained by modal conversion on second time sequence data, the second time sequence data being time sequence data other than the first time sequence data in the plurality of time sequence data; and a training module, configured to train a time sequence feature extraction model according to the positive sample pairs and the negative sample pairs to obtain a trained time sequence feature extraction model, the time sequence feature extraction model being configured to extract a feature vector of time sequence data.

[0021] In a possible implementation, the positive sample pair further includes first time-series enhancement data and second time-series enhancement data, and the negative sample pair further includes the first time-series enhancement data and third time-series enhancement data; the first time-series enhancement data is time-series data obtained by enhancing the first sub-time-series data by using a first enhancement manner, the second time-series enhancement data is time-series data obtained by enhancing the second sub-time-series data by using the first enhancement manner, and the third time-series enhancement data is time-series data obtained by enhancing the first sub-time-series data or the second sub-time-series data by using a second enhancement manner; the first sub-time-series data and the second sub-time-series data are two different sub-time-series data in the first time-series data, and the first time-series data is any time-series data in the plurality of time-series data.

[0022] In a possible implementation, the training module is further configured to train the visual feature extraction model according to the positive sample pair and the negative sample pair, to obtain the trained visual feature extraction model, and the visual feature extraction model is configured to extract a feature vector of the image data.

[0023] In a possible implementation, the visual feature extraction model is a model obtained by training a visual base model by using the plurality of image data, and the visual feature extraction model has the ability to extract a structural information representation in the image data.

[0024] In a fourth aspect, a time-series data processing apparatus is provided, configured to execute the method in the second aspect or any possible implementation of the second aspect. Specifically, the time-series data processing apparatus includes modules configured to execute the method in the second aspect or any possible implementation of the second aspect.

[0025] In a possible implementation, the apparatus includes: a first acquisition module configured to acquire time-series data to be processed; an extraction module configured to extract a feature vector of the time-series data based on a feature extraction model, the feature extraction model including a time-series feature extraction model and / or a visual feature extraction model, the time-series feature extraction model being a time-series feature extraction model trained by the training method in the first aspect, and the visual feature extraction model being a visual feature extraction model trained by the training method in the first aspect; and a second acquisition module configured to acquire a processing result of the time-series data according to the feature vector of the time-series data.

[0026] In a possible implementation, the time-series data to be processed belongs to a first task scenario; the apparatus further includes: a fine-tuning module configured to fine-tune the feature extraction model by using a plurality of first time-series data in the first task scenario, to obtain a fine-tuned feature extraction model; and the extraction module is configured to acquire the feature vector of the time-series data by using the fine-tuned feature extraction model.

[0027] In a possible implementation, the fine-tuning module is configured to acquire a plurality of first sample pairs based on the plurality of first time-series data, the plurality of first sample pairs including positive first sample pairs and negative first sample pairs; and fine-tune the feature extraction model according to the positive first sample pairs and the negative first sample pairs to obtain a fine-tuned feature extraction model.

[0028] In a possible implementation, the extraction module is configured to invoke the feature extraction model to acquire a feature vector of the time-series data.

[0029] In a possible implementation, the second acquisition module is configured to acquire a processing result of the time-series data according to the feature vector of the time-series data and a task network, the task network including a classification network, a prediction network, or a regression network.

[0030] In a fifth aspect, a computer device is provided, which includes a processor coupled with a memory, and the memory stores at least one program instruction or code, which is loaded and executed by the processor to enable the computer device to implement the training method of the feature extraction model according to the first aspect, or to enable the computer device to implement the processing method of the time-series data according to the first aspect.

[0031] Optionally, the processor is one or more, and the memory is one or more.

[0032] Optionally, the memory can be integrated with the processor, or the memory and the processor are separately arranged.

[0033] In the implementation process, the memory can be a non-transitory memory, for example, a read only memory (ROM), which can be integrated on the same chip with the processor, or can be separately arranged on different chips, and the type of the memory and the arrangement manner of the memory and the processor are not limited in the present application.

[0034] In a sixth aspect, a computer readable storage medium is provided, which stores at least one instruction, and the instruction is loaded and executed by a processor to enable a computer to implement the method in the first aspect or any possible implementation of the first aspect, or to implement the method in the second aspect or any possible implementation of the second aspect.

[0035] In a seventh aspect, a computer program (product) is provided, which includes computer program code, and when the computer program code is run by a computer, the computer is enabled to execute the method in the aspects.

[0036] In an eighth aspect, a chip is provided, comprising a processor configured to invoke and run instructions stored in a memory, so that a communication device in which the chip is installed performs the method in any of the aspects above.

[0037] In a ninth aspect, another chip is provided, comprising an input interface, an output interface, a processor and a memory, the input interface, the output interface, the processor and the memory are connected through internal connection paths, the processor is configured to execute code in the memory, when the code is executed, the processor is configured to perform the method in any of the aspects above.

[0038] It should be understood that the beneficial effects achieved by the third aspect to the ninth aspect of the present application and the corresponding possible implementation manners can be referred to the technical effects of the first aspect to the second aspect and the corresponding possible implementation manners described above, and will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0039] FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0040] FIG. 2 is a flowchart of a feature extraction model training method provided by an embodiment of the present application;

[0041] FIG. 3 is a schematic diagram of a model training process provided by an embodiment of the present application;

[0042] FIG. 4 is a schematic diagram of another model training process provided by an embodiment of the present application;

[0043] FIG. 5 is a schematic diagram of another model training process provided by an embodiment of the present application;

[0044] FIG. 6 is a flowchart of a time series data processing method provided by an embodiment of the present application;

[0045] FIG. 7 is a flowchart of a training method and a processing method provided by an embodiment of the present application;

[0046] FIG. 8 is a structural schematic diagram of a feature extraction model training apparatus provided by an embodiment of the present application;

[0047] FIG. 9 is a structural schematic diagram of a time series data processing apparatus provided by an embodiment of the present application;

[0048] FIG. 10 is a structural schematic diagram of a computer device provided by an embodiment of the present application;

[0049] FIG. 11 is a structural schematic diagram of a server provided by an embodiment of the present application. DETAILED DESCRIPTION

[0050] For the purposes of the present application, the technical solutions and advantages will be more apparent in the following detailed description of the embodiments of the present application with reference to the accompanying drawings.

[0051] With the increasingly wide application of time series data processing in various fields, the modeling method for time series data has developed rapidly. However, there are still many problems in the application of actual scenarios, mainly divided into two categories: first, the amount of time series data in the actual scene is too small, which cannot meet the data amount required for training the model, that is, the local point rapid start problem, or the cold start problem; second, the model performance can be further improved, for example, the classification accuracy of time series data can be further improved.

[0052] In the related art, a case by case modeling method is usually used to train the model, that is, for each task scenario, the model is trained according to a large amount of time series data of the current task scenario to obtain a feature extraction model strongly related to the current business scenario. However, for the task scenarios with less samples or annotations in the time series field, since the amount of time series data that can be used for model training is also small, it is difficult to meet the model training needs in the related art. And since the feature extraction model obtained in the related art is strongly related to the current business scenario, the generalization performance of the feature extraction model is poor, and it is difficult to solve the cold start problem of the above-mentioned task scenarios with less samples or annotations.

[0053] Exemplarily, taking the multi-dimensional time series classification problem as an example, the vision time series transformer (ViTST) model can solve the classification problem in the time series field based on the pre-training model in the image field. By fine-tuning the image pre-training model, the ability of the image encoder to perceive shapes can be released, thereby completing the mapping of the original time series data space to the image data space. The ViTST model includes a data preprocessing module, a pre-training image encoder information extraction module, and a classification prediction head.

[0054] The data preprocessing module: the time series data is plotted point by point to convert the time series data into a corresponding connected graph, i.e., a time series image; the pre-trained image encoder information extraction module: a pre-trained basic model in an image field is selected, for example, a vision transformer (ViT) model, the model parameters of the ViT model are fixed or not fixed, the ViT model participates in fine-tuning, and the related information of the time series image is extracted through the vision encoder; the classification prediction head: the related information extracted by the vision encoder is mapped to a corresponding category label, the model is trained based on supervised learning, and a high-precision classification model is obtained. Therefore, the ViTST model predicts the category label of the time series in the same way as image classification through the vision encoder, and it is proved that the knowledge of the vision model pre-trained from the image can be successfully transferred to the visualized time series image.

[0055] However, the ViTST model based on the pre-trained vision encoder focuses on the connected shape information of the time series data, and ignores the data periodicity, trend and other time series representations covered by the time series mode itself, so that the information extraction is incomplete and has certain limitations. And the ViTST model needs a large amount of labeled time series data for full-amount supervised learning modeling, and high-quality labels often have a great acquisition cost, which cannot solve the cold start problem.

[0056] The feature extraction model training method and the time series data processing method provided in the embodiments of the present application can be applied to various fields such as energy, transportation, economy, weather, medicine, nature and society, for example, power generation prediction, climate prediction, electrocardiogram diagnosis, fault diagnosis (such as wireless network fault diagnosis and fan fault diagnosis), motion posture analysis, human posture recognition, spectrum classification and other task scenarios. FIG. 1 shows a schematic diagram of an implementation environment provided by the embodiments of the present application. The implementation environment can include a terminal 11 and a server 12. The terminal 11 is installed with an application program or a webpage having a time series data processing function, and the server 12 can provide background services for the application program or the webpage installed by the terminal.

[0057] The feature extraction model training method provided in the embodiments of the present application can be executed by the terminal 11, or by the server 12, or by the terminal 11 and the server 12 together, which is not limited in the embodiments of the present application. For the case that the feature extraction model training method provided in the embodiments of the present application is executed by the terminal 11 and the server 12 together, the server 12 undertakes the main computing work, and the terminal 11 undertakes the secondary computing work; or the server 12 undertakes the secondary computing work, and the terminal 11 undertakes the main computing work; or the server 12 and the terminal 11 adopt a distributed computing architecture for collaborative computing.

[0058] The processing method of time series data provided in the embodiments of the present application can be executed by the terminal 11, can also be executed by the server 12, and can also be executed by the terminal 11 and the server 12 together, which is not limited in the embodiments of the present application. For the case that the processing method of time series data provided in the embodiments of the present application is executed by the terminal 11 and the server 12 together, the server 12 undertakes the main computing work, and the terminal 11 undertakes the secondary computing work; or the server 12 undertakes the secondary computing work, and the terminal 11 undertakes the main computing work; or the server 12 and the terminal 11 adopt a distributed computing architecture to perform collaborative computing.

[0059] It should be noted that the execution device of the feature extraction model training method and the execution device of the processing method of time series data can be the same or different, which is not limited in the embodiments of the present application.

[0060] Optionally, the terminal 11 can be any electronic product that can interact with a user through one or more of a keyboard, a touchpad, a touch screen, a remote controller, voice interaction, or a handwriting device, such as a PC (personal computer), a mobile phone, a smart phone, a PDA (personal digital assistant), a wearable device, a PPC (pocket PC), a tablet computer, a smart car machine, a smart television, a smart sound box, a smart voice interaction device, a smart home appliance, a vehicle terminal, a VR (virtual reality) device, an AR (augmented reality) device, and the like. The server 12 can be a server or a server cluster composed of multiple servers, or a cloud computing service center. The terminal 11 and the server 12 establish a communication connection through a wired or wireless network.

[0061] Those skilled in the art should understand that the terminal 11 and the server 12 described above are only examples, and other existing or future terminal or server, such as those applicable to the present application, should also be included in the protection scope of the present application and are hereby included by reference.

[0062] Referring to FIG. 2, FIG. 2 is a flowchart of a feature extraction model training method provided in the embodiments of the present application. The feature extraction model training method is executed by a computer device, which can be the terminal 11 shown in FIG. 1 or the server 12 shown in FIG. 1, which is not limited in the embodiments of the present application. As shown in FIG. 2, the feature extraction model training method provided in the embodiments of the present application can include the following steps 201 to 203.

[0063] Step 201, obtaining a plurality of time series data, the plurality of time series data being derived from a plurality of task scenarios.

[0064] The time series data refers to a series of data obtained at different time periods or time points in chronological order, used to describe the situation or degree of the observation object changing with time. One time series data usually includes multiple time stamps and observation values corresponding to each time stamp. In the embodiments of the present application, the time series data used for model training can be collected from different task scenarios in various fields such as energy, transportation, economy, weather, medicine, nature and society, to realize model training based on large-scale multi-field data sets. Thus, the model trained by the time series data of multiple task scenarios can be applied to various business scenarios and has good generalization performance. Optionally, the dimensions of the multiple time series data can be one-dimensional, two-dimensional or any other dimension.

[0065] In step 202, a plurality of sample pairs are obtained based on a plurality of time series data, and the plurality of sample pairs include positive sample pairs and negative sample pairs.

[0066] After obtaining the plurality of time series data, at least one sample pair corresponding to each time series data can be obtained. Each sample pair includes two samples. If the two samples belong to the same distribution of data points, they are positive sample pairs. If the two samples belong to different distributions of data points, they are negative sample pairs. For example, similar to clustering, the two samples in the positive sample pair belong to the same class, and the two samples in the negative sample pair belong to different classes. The embodiments of the present application do not limit the way of obtaining sample pairs. The obtained positive and negative samples include but are not limited to the following three cases.

[0067] Case one: the positive sample pair includes first time series data and first image data, and the negative sample pair includes first time series data and second image data; the first time series data is any time series data in the plurality of time series data, the first image data is image data obtained by modal conversion of the first time series data, and the second image data is image data obtained by modal conversion of the second time series data; the second time series data is time series data other than the first time series data in the plurality of time series data.

[0068] In this case, for any time series data in the plurality of time series data, the any time series data is converted in modal to generate image data (also referred to as time series graph) corresponding to the any time series data. Optionally, the time series data can be converted into image data by various methods, including but not limited to wavelet transform, Gabor angular field, Markov transition field, recursive graph or connection graph, etc. For example, taking time series data [t1, t2, t3,..., tn] as an example, N is a positive integer greater than 3, t1, t2, t3,..., tn are plotted on the coordinate axis as coordinate points, and the coordinate points are connected to obtain image data in image format. Wherein, the abscissa represents the time point, and the ordinate represents the observation value. In the embodiments of the present application, any time series data and image data corresponding to any time series data are taken as positive sample pairs, and any time series data and image data corresponding to other time series data than any time series data are taken as negative sample pairs.

[0069] In the above modal conversion process, for single-dimensional time series data, direct conversion is performed; for multi-dimensional time series data, each dimension of the multi-dimensional time series data can be converted into corresponding image data, and then each dimension of the image data can be spliced to obtain image data corresponding to the multi-dimensional time series data. In the embodiments of the present application, the sizes of the image data converted from the plurality of time series data are the same, for example, the size of the image data is 256*256. Optionally, the image data can be a black and white image, or a color image, for example, an RGB image. RGB respectively refers to red (R), green (G), and blue (B). In the case where the image data includes image data corresponding to multiple dimensions, the colors of the image data corresponding to the multiple dimensions can be different, so as to better distinguish the time series data of different dimensions.

[0070] Case two, the positive sample pair includes first time series enhanced data and second time series enhanced data, and the negative sample pair includes the first time series enhanced data and third time series enhanced data; the first time series enhanced data is time series data obtained by enhancing the first sub time series data by using the first enhancement manner, the second time series enhanced data is time series data obtained by enhancing the second sub time series data by using the first enhancement manner, and the third time series enhanced data is time series data obtained by enhancing the first sub time series data or the second sub time series data by using the second enhancement manner; the first sub time series data and the second sub time series data are two different sub time series data in the first time series data.

[0071] In this case two, for any time series data in the plurality of time series data, for each dimension of time series data in any time series data, each dimension of time series data is segmented by random cropping to obtain two sub time series data, which belong to different views of the same time series data. For the two sub time series data obtained by segmentation, different data enhancement methods are used for enhancement, including but not limited to inversion, rotation, slicing, scaling, reordering, etc. The time series enhancement data obtained by the same data enhancement method has stronger connection, which is called positive sample pair, while the time series enhancement data generated by different data enhancement methods has limited correlation, which is called negative sample pair.

[0072] Case three, the positive sample pair includes the first time series data and the first image data, and the negative sample pair includes the first time series data and the second image data; and the positive sample pair further includes the first time series enhancement data and the second time series enhancement data, and the negative sample pair further includes the first time series enhancement data and the third time series enhancement data.

[0073] In this case three, for any time series data in the plurality of time series data, both include the time series-image multimodal sample pair in case one, that is, the positive sample pair includes the first time series data and the first image data, and the negative sample pair includes the first time series data and the second image data; and the multi-view enhancement sample pair in case two, that is, the positive sample pair further includes the first time series enhancement data and the second time series enhancement data, and the negative sample pair further includes the first time series enhancement data and the third time series enhancement data. The acquisition method of the sample pair in case three can refer to the acquisition method of the sample pair in case one and case two, which will not be described here.

[0074] Step 203, training the time series feature extraction model according to the positive sample pair and the negative sample pair to obtain the trained time series feature extraction model, and the time series feature extraction model is used to extract the feature vector of the time series data.

[0075] After obtaining the positive and negative sample pairs, the time series feature extraction model can be trained according to the positive and negative sample pairs, for example, by using the contrast learning method to unsupervisedly train the feature extraction model. Contrast learning is to extract meaningful representation by comparing positive and negative samples. Contrast learning is used to minimize the similarity between positive sample pairs and maximize the similarity between negative sample pairs in the feature representation space, so that the potential differences between different categories of samples can be learned without labeling.

[0076] In the embodiments of the present application, since the conditions of the obtained positive and negative sample pairs are different, the ways of training the time series feature extraction model according to the positive and negative sample pairs are also different.

[0077] The training manner one for the case one can be that, a time sequence feature extraction model is called to extract feature vectors of time sequence data in the plurality of sample pairs, and a visual feature extraction model is called to extract feature vectors of image data in the plurality of sample pairs; a contrast loss is obtained according to similarities between the feature vectors of the positive sample pairs and similarities between the feature vectors of the negative sample pairs; model parameters of the time sequence feature extraction model are updated based on the contrast loss to obtain an updated time sequence feature extraction model; it is determined whether the updated time sequence feature extraction model meets a training termination condition; if the updated time sequence feature extraction model meets the training termination condition, the updated time sequence feature extraction model is taken as the trained time sequence feature extraction model.

[0078] If the updated time sequence feature extraction model does not meet the training termination condition, the visual feature extraction model and the updated time sequence feature extraction model are called to obtain a new contrast loss, and the model parameters of the updated time sequence feature extraction model are updated again by using the new contrast loss. This is repeated until a time sequence feature extraction model meeting the training termination condition is obtained, and the time sequence feature extraction model meeting the training termination condition is taken as the trained time sequence feature extraction model. In the contrast learning, the learning goal of the contrast loss is to make the similarities between the feature vectors of the positive sample pairs greater and the similarities between the feature vectors of the negative sample pairs smaller. Optionally, the similarities can be measured by indicators such as Euclidean distance, Manhattan distance, Chebyshev distance, cosine similarity or Pearson correlation coefficient.

[0079] The training termination condition can be set according to experience or flexibly adjusted according to application scenarios, and embodiments of the present application do not limit this. In an exemplary embodiment, the time sequence feature extraction model meeting the training termination condition includes but is not limited to any one of the following: the number of model parameter updates (which can also be referred to as the number of iterations) performed when obtaining the trained time sequence feature extraction model reaches a maximum number of iterations, the contrast loss on which the trained time sequence feature extraction model is based converges, and the contrast loss on which the trained time sequence feature extraction model is based is less than a loss threshold. The maximum number of iterations and the loss threshold are set according to experience or flexibly adjusted according to application scenarios, and embodiments of the present application do not limit this.

[0080] The embodiments of the present application do not limit the model architecture of the time series feature extraction model or the visual feature extraction model, and can be any model capable of feature extraction, including but not limited to an encoder, a convolutional neural network, a deep neural network, a random forest algorithm, a logistic regression algorithm, or a support vector machine (SVM), etc. In a possible implementation, the visual feature extraction model is a model obtained by training a visual base model with multiple image data, i.e., a mature visual model such as a ViT model trained with massive image data, so that the visual feature extraction model has the ability to extract structural information representation in image data. Optionally, the visual feature extraction model also has the ability to extract color features, texture features, shape features, and spatial relationship features in image data.

[0081] Exemplarily, taking the time series feature extraction model as a time series encoder (TS Encoder) and the visual feature extraction model as an image encoder (Image Encoder) as an example, referring to the schematic diagram of the model training process shown in FIG. 3. For N*C time series data, the image data corresponding to each time series data is obtained, C is the dimension of the time series data, and N is the number of time series data. The time series data is input into the TS Encoder, and the feature vectors of the time series data, such as T1, T2, …, TN, are extracted via the TS Encoder; the image data is input into the Image Encoder, and the feature vectors of the image data, such as I1, I2, …, IN, are extracted via the Image Encoder. Wherein, the TS Encoder or the Image Encoder is connected with a project head, which is used to obtain the output feature vectors by using the features extracted by the TS Encoder or the Image Encoder. In the case of multi-dimensional time series data, the time series data is dimensionally split, and the feature vectors of the time series data of each dimension are obtained by sequentially inputting the time series data of each dimension into the TS Encoder, and the feature vectors of the time series data of each dimension are fused to obtain the feature vectors of the overall multi-dimensional time series data.

[0082] Afterwards, the time series data and its corresponding image data are taken as positive sample pairs, and the time series data and the image data corresponding to other time series data are taken as negative sample pairs to form time series-image multi-modal sample pairs. Based on multi-modal contrast learning, the TS Encoder model has global modeling capability of the time series modality and obtains the extraction capability of the structural information representation of the image modality. In the case of the Image Encoder being a pre-trained ViT model, the TS Encoder can be trained by using the ViT model to guide the training of the TS Encoder by means of knowledge distillation, so that the model training process shown in FIG. 3 can be referred to as image knowledge distillation.

[0083] The training mode two for the above case two can be that a time series feature extraction model is called to extract feature vectors of the time series enhanced data in the multiple sample pairs; a contrast loss is obtained according to the similarity between the feature vectors of the positive sample pairs and the similarity between the feature vectors of the negative sample pairs; model parameters of the time series feature extraction model are updated based on the contrast loss to obtain an updated time series feature extraction model; it is determined whether the updated time series feature extraction model meets a training termination condition; and if the updated time series feature extraction model meets the training termination condition, the updated time series feature extraction model is taken as the trained time series feature extraction model.

[0084] If the updated time series feature extraction model does not meet the training termination condition, a new contrast loss is obtained by calling the updated time series feature extraction model, and the model parameters of the updated time series feature extraction model are updated again by using the new contrast loss. In this way, the training is repeated until a time series feature extraction model meeting the training termination condition is obtained, and the time series feature extraction model meeting the training termination condition is taken as the trained time series feature extraction model. The training termination condition and other contents in the training mode two can be referred to the training mode one, which will not be described herein again.

[0085] Exemplarily, still taking the time series feature extraction model as the TS Encoder as an example, referring to the schematic diagram of the model training process shown in FIG. 4. For any time series data, random crop is used to divide the time series data into two sub-time series data; the multi-aug staking module is used to enhance the two sub-time series data by using multiple enhancement methods to obtain time series enhanced data corresponding to the multiple enhancement methods respectively; and the TS Encoder and the project head are used to obtain feature vectors of the multiple time series enhanced data corresponding to the two sub-time series data respectively.

[0086] Afterwards, the time sequence enhancement data of the two sub time sequence data obtained by using the same enhancement manner are taken as the positive sample pair, the time sequence enhancement data of the two sub time sequence data obtained by using different enhancement manners are taken as the negative sample pair, and a multi-view multi-enhancement manner mixed sample pair is constructed, that is, a multi-view multi-enhancement manner mixed contrast learning paradigm is constructed. Therefore, the model training process shown in FIG. 4 can be called multi-view enhanced adaptive mixed contrast learning. In the process of updating the model parameters, the contrast learning weights between all sample pairs can be updated in a sample data driven manner, for example, the contrast learning weights between all sample pairs are updated based on the neural network in a sample data driven manner; and then the model parameters are updated and optimized according to the contrast learning weights and the contrast loss, so as to realize the adaptive learning strategy of dynamically adjusting the importance of the positive sample pair. Alternatively, the contrast learning weights between all sample pairs can also be updated by manually setting hyperparameters.

[0087] For the training manner three of the above case three, the time sequence feature extraction model can be trained based on the positive and negative sample pairs including the time sequence data and the image data, to obtain a time sequence feature extraction model after one round of training. Then the time sequence feature extraction model after one round of training is trained based on the positive and negative sample pairs including the time sequence enhancement data, to obtain a trained time sequence feature extraction model. Alternatively, the time sequence feature extraction model after one round of training is trained based on the positive and negative sample pairs including the time sequence enhancement data, to obtain a time sequence feature extraction model after one round of training; and then the time sequence feature extraction model is trained based on the positive and negative sample pairs including the time sequence data and the image data, to obtain a trained time sequence feature extraction model. Alternatively, the time sequence feature extraction model is trained based on the positive and negative sample pairs including the time sequence data and the image data and the positive and negative sample pairs including the time sequence enhancement data, to obtain a trained time sequence feature extraction model.

[0088] In the training manner three, the model training manner based on the positive and negative sample pairs including the time sequence data and the image data can refer to the training manner one, and the model training manner based on the positive and negative sample pairs including the time sequence enhancement data can refer to the training manner two, which will not be described herein again.

[0089] In the embodiments of the present application, for the case that the positive and negative sample pairs include the image data in the above case one and case three, in addition to training the time sequence feature extraction model according to the positive and negative sample pairs to obtain a trained time sequence feature extraction model, the visual feature extraction model can also be trained according to the positive and negative sample pairs to obtain a trained visual feature extraction model, and the visual feature extraction model is used to extract the feature vector of the image data.

[0090] Thus, the training manner one or the training manner three for the model training manner including the time series data and the image data can be changed to: calling the time series feature extraction model to extract the feature vectors of the time series data in the plurality of sample pairs, and calling the visual feature extraction model to extract the feature vectors of the image data in the plurality of sample pairs; obtaining the contrast loss according to the similarity between the feature vectors of the positive sample pairs and the similarity between the feature vectors of the negative sample pairs; updating the model parameters of the time series feature extraction model and the visual feature extraction model based on the contrast loss to obtain the updated time series feature extraction model and the updated visual feature extraction model; determining whether the updated time series feature extraction model and the updated visual feature extraction model satisfy the training termination condition; and if the updated time series feature extraction model and the updated visual feature extraction model satisfy the training termination condition, taking the updated time series feature extraction model as the trained time series feature extraction model and taking the updated visual feature extraction model as the trained visual feature extraction model.

[0091] If the updated time series feature extraction model and the updated visual feature extraction model do not satisfy the training termination condition, the new contrast loss is obtained by calling the updated time series feature extraction model and the updated visual feature extraction model, and the model parameters of the updated time series feature extraction model and the updated visual feature extraction model are updated again by using the new contrast loss. In this way, the time series feature extraction model and the visual feature extraction model satisfying the training termination condition are obtained, the time series feature extraction model satisfying the training termination condition is taken as the trained time series feature extraction model, and the visual feature extraction model satisfying the training termination condition is taken as the trained visual feature extraction model.

[0092] Exemplarily, taking the time series feature extraction model as TS Encoder and the visual feature extraction model as Image Encoder as an example, FIG. 5 is a schematic diagram of another model training process provided by an embodiment of the present application. The model training process shown in FIG. 5 includes the image knowledge distillation module shown in FIG. 3 and the multi-view enhanced adaptive hybrid contrast learning module shown in FIG. 4, and the order of training the time series feature extraction model and the multi-view enhanced adaptive hybrid contrast learning module is not limited in the embodiment of the present application. The implementation of the model training process shown in FIG. 5 can refer to the related description of FIG. 3 and FIG. 4, which will not be repeated here.

[0093] In summary, the pre-training of the feature extraction model is achieved through the above steps 201-203. Through multi-modal contrast learning, the visual feature extraction model can effectively improve the ability of the time series feature extraction model to extract pattern features of time series data, i.e., the ability to represent structural information. Through multi-view data augmentation to increase the diversity of time series data and adaptive mixed contrast learning under different views, the time series feature extraction model can learn common information and special information under different views, thereby improving the model's domain adaptation ability, enhancing the model's performance, and alleviating the cold start problem.

[0094] The method provided by the embodiments of the present application trains the time series feature extraction model through time series data in a multi-task scenario, so that the trained time series feature extraction model can be applied to various business scenarios and has good generalization performance. Unsupervised contrast learning is used in the training process, avoiding the dependence on annotations. Moreover, the time series feature extraction model is trained through positive and negative sample pairs of time series data and image data, so that the trained time series feature extraction model can learn not only sequential representations such as periodicity and trend of time series modal data, but also pattern representations such as structure and shape of image modal data, improving the feature extraction ability of the trained time series feature extraction model and solving the cold start problem of task scenarios with less samples or annotations. The time series feature extraction model is trained through positive and negative sample pairs of time series enhancement data in multiple enhancement modes under multiple views, so that the trained time series feature has stronger generalization ability.

[0095] Referring to FIG. 6, FIG. 6 is a flowchart of a time series data processing method provided by an embodiment of the present application. The time series data processing method is executed by a computer device, which can be the terminal 11 shown in FIG. 1 or the server 12 shown in FIG. 1, and the embodiments of the present application do not limit this. As shown in FIG. 6, the time series data processing method provided by the embodiments of the present application can include the following steps 601-603.

[0096] Step 601: Obtain time series data to be processed.

[0097] In the embodiments of the present application, the time series data to be processed belongs to a first task scenario, which can also be referred to as a downstream task scenario. Optionally, the first task scenario can be any of the multiple task scenarios in step 201, or a task scenario other than the multiple task scenarios in step 201. For example, the first task scenario can be electrocardiogram diagnosis or human posture recognition.

[0098] The time series data to be processed can be any one of the plurality of time series data collected in advance in the first task scenario, or can be online time series data acquired in real time. For example, in an electrocardiogram diagnosis scenario, the time series data to be processed can be an electrocardiogram collected in real time by an electrocardiograph.

[0099] In step 602, a feature vector of the time series data is extracted based on a feature extraction model. The feature extraction model includes a time series feature extraction model and / or a visual feature extraction model. The time series feature extraction model is trained by the training method shown in FIG. 2, and the visual feature extraction model is trained by the training method shown in FIG. 2.

[0100] In the examples of the present application, based on the training method shown in FIG. 2, the pre-training of the feature extraction model is realized by the time series data in multiple fields, so that the pre-trained feature extraction model has a certain feature extraction capability. Further, the pre-trained feature extraction model can be used to extract the feature vector of the time series data to be processed. Optionally, the pre-trained feature extraction model can be directly called to extract the feature vector of the time series data, or the pre-trained feature extraction model can be fine-tuned by the plurality of first time series data in the first task scenario to obtain a fine-tuned feature extraction model; and the fine-tuned feature extraction model is called to obtain the feature vector of the time series data.

[0101] In a possible implementation, the way of fine-tuning the feature extraction model by the plurality of first time series data in the first task scenario to obtain the fine-tuned feature extraction model can include: obtaining a plurality of first sample pairs based on the plurality of first time series data, the plurality of first sample pairs including positive first sample pairs and negative first sample pairs; and fine-tuning the feature extraction model according to the positive first sample pairs and the negative first sample pairs to obtain the fine-tuned feature extraction model. The implementation of obtaining the plurality of first sample pairs based on the plurality of first time series data can refer to the implementation of obtaining the plurality of sample pairs based on the plurality of time series data in step 202; and the implementation of fine-tuning the feature extraction model according to the positive first sample pairs and the negative first sample pairs can refer to the implementation of training the time series feature extraction model according to the positive sample pairs and the negative sample pairs in step 203, which will not be described herein again.

[0102] In the above fine-tuning process, a task network can be added behind the feature extraction model to realize the mapping of the feature vector to the label through the task network. The task network can be flexibly selected according to the downstream task scenario. For example, in the classification scenario of the time series data, the label is a category, and the task network is a classification network, such as a fully connected neural network, a convolutional neural network, or a recurrent neural network. In the continuity prediction scenario of the time series data, the label is a prediction result, and the task network is a prediction network or a regression network, such as a time series model, an artificial neural network, or a grey prediction model.

[0103] Optionally, for the scene that the feature extraction model includes a time sequence feature extraction model and a visual feature extraction model, the first feature vector output by the time sequence feature extraction model and the second feature vector output by the visual feature extraction model are fused to obtain a fused feature vector, and the task network is used to realize mapping of the fused feature vector to a label. The embodiments of the present application do not limit the fusion manner of the feature vector, including but not limited to average fusion, weighted average fusion, splicing fusion, maximum / minimum fusion, self-attention mechanism fusion, neural network fusion or deep learning fusion, etc.

[0104] In the scene of adding the task network, the plurality of first time sequence data for fine-tuning is labeled time sequence data, and the label is the correct processing result corresponding to the time sequence data. Therefore, the feature extraction model is fine-tuned according to the positive first sample pair and the negative first sample pair to obtain a fine-tuned feature extraction model, including: fine-tuning the feature extraction model and the task network according to the positive first sample pair, the negative first sample pair and the label to obtain the fine-tuned feature extraction model and the fine-tuned task network; or fine-tuning the feature extraction model according to the positive first sample pair and the negative first sample pair to obtain the fine-tuned feature extraction model, calling the fine-tuned feature extraction model, training the task network based on the plurality of first time sequence data and the label to obtain a trained task network.

[0105] In step 603, the processing result of the time sequence data is obtained according to the feature vector of the time sequence data.

[0106] In the embodiments of the present application, since the feature vector extracted based on the feature extraction model can represent the feature information of the time sequence data, the time sequence data can be processed according to the similarity between the feature vectors, and the inference in the actual application process is realized. For example, the category to which the time sequence data belongs is determined, or the time sequence data is predicted, or the time sequence data is diagnosed for fault, etc.

[0107] For the case that the task network is fine-tuned or trained in the fine-tuning process, the processing result of the time sequence data can be obtained according to the feature vector of the time sequence data and the task network. For different task scenes, the task network includes a classification network, a prediction network or a regression network, and the task network is the fine-tuned task network or the trained task network in step 602.

[0108] Exemplarily, the overall process of the training method and the processing method provided by the embodiments of the present application is summarized in combination with FIG. 7. The overall process includes three stages of pre-training, fine-tuning and inference. For the processing task of time series data in different business scenarios, the pre-training in the first stage is decoupled from the downstream task. The pre-training generally refers to pre-training the model through massive multi-domain data before the downstream task, so as to obtain a basic model that has learned corresponding knowledge. Fine-tuning refers to adjusting the model parameters using a certain amount of current task scenario data for the specified downstream task scenario, so that the model learns some information contained in the current task, thereby improving the accuracy of the current task.

[0109] As shown in FIG. 7, in the pre-training stage, a feature extraction model is trained according to the time series data set of the multi-task scenario. The feature extraction model includes an image knowledge distillation module and a multi-view enhanced adaptive hybrid contrast learning module. The image knowledge distillation module extracts the structural information representation of the image data corresponding to the time series data based on the trained visual feature extraction model, constructs information alignment interaction between different modalities of the same time series data, and enables the time series feature extraction model to acquire the ability of mode and structural information extraction through multi-modal contrast learning. The multi-view enhanced adaptive hybrid contrast learning module can enhance the generalization of the model by multi-view segmentation and multiple data augmentation of the same time series data, and adaptively learn the learning weight between different positive sample pairs in the contrast learning process in a data-driven manner, so that the feature extraction model can learn the common information and special information in different views, thereby improving the domain adaptation ability of the feature extraction model and enhancing the performance of the model in processing time series data and alleviating the local cold start problem.

[0110] In the fine-tuning process of the second stage, the pre-trained feature extraction model is fine-tuned using the time series data set of the downstream task scenario. The model fine-tuning process is consistent with the pre-training process, so that the fine-tuned feature extraction model not only captures multi-domain general knowledge, but also focuses on the knowledge in the specific downstream task domain, thereby improving the processing capability of the model in the specific task scenario. The high-precision processing model can be obtained according to the fine-tuned feature extraction model. The high-precision processing model can only include the image knowledge distillation module, only include the multi-view enhanced adaptive hybrid contrast learning module, or include both the image knowledge distillation module and the multi-view enhanced adaptive hybrid contrast learning module. In the inference process of the third stage, the processing result of the online input time series data is obtained based on the high-precision processing model.

[0111] The test results of the feature extraction model provided in the embodiments of the present application are introduced taking the multiple first time series data in the first task scenario as the time series data in the UCR open source data set. The UCR open source data set is a time series data set, which contains 130 data sets including vehicle engine defect detection, and each data set includes multiple labeled time series data. The pre-trained feature extraction model is fine-tuned by 80% and 100% of the data in the UCR open source data set in the embodiments of the present application, and is compared with the model in the related art, and the evaluation index is accuracy (Accuracy).

[0112] Table 1

[0113] The actual test results are shown in Table 1. In the case of fine-tuning based on 80% data, i.e. the present application (80%), the accuracy of the fine-tuned feature extraction model in the embodiments of the present application has reached a high precision, which can alleviate the cold start problem. In the case of fine-tuning based on 100% data, i.e. the present application (100%), the fine-tuned feature extraction model in the embodiments of the present application has reached the state-of-the-art (SOTA) effect, i.e. the highest accuracy of 0.857.

[0114] In summary, the processing method provided in the embodiments of the present application uses the feature extraction model trained by the training method shown in the above FIG. 2 to extract the feature vector of the time series data to be processed. Since the training method shown in the above FIG. 2 has high feature extraction capability, the extracted feature vector of the time series data to be processed is more accurate and can better represent the features of the time series data to be processed, and thus the processing result obtained according to the feature vector is more accurate.

[0115] The training method of the feature extraction model and the processing method of the time series data of the embodiments of the present application are introduced above. Corresponding to the above method, the embodiments of the present application also provide a training device of a feature extraction model and a processing device of time series data. FIG. 8 is a structural schematic diagram of a training device of a feature extraction model provided in the embodiments of the present application, which is applied to a computer device for executing the method shown in the above FIG. 2. Based on the following multiple modules shown in FIG. 8, the training device of the feature extraction model shown in FIG. 8 can perform all or part of the operations performed by the computer device. It should be understood that the device can include more additional modules than the shown modules or omit part of the shown modules, and the embodiments of the present application do not limit this. As shown in FIG. 8, the device includes:

[0116] The acquisition module 801 is configured to acquire multiple time series data, and the multiple time series data are derived from multiple task scenarios.

[0117] The acquisition module 801 is further configured to acquire a plurality of sample pairs based on the plurality of time sequence data, the plurality of sample pairs including positive sample pairs and negative sample pairs, the positive sample pairs including first time sequence data and first image data, and the negative sample pairs including the first time sequence data and second image data; the first time sequence data is any time sequence data in the plurality of time sequence data, the first image data is image data obtained by performing modality conversion on the first time sequence data, and the second image data is image data obtained by performing modality conversion on second time sequence data, the second time sequence data being time sequence data other than the first time sequence data in the plurality of time sequence data.

[0118] The training module 802 is configured to train a time sequence feature extraction model according to the positive sample pairs and the negative sample pairs, to obtain a trained time sequence feature extraction model, and the time sequence feature extraction model is configured to extract a feature vector of time sequence data.

[0119] In a possible implementation, the positive sample pairs further include first time sequence enhancement data and second time sequence enhancement data, and the negative sample pairs further include the first time sequence enhancement data and third time sequence enhancement data; the first time sequence enhancement data is time sequence data obtained by performing enhancement on first sub-time sequence data by using a first enhancement manner, the second time sequence enhancement data is time sequence data obtained by performing enhancement on second sub-time sequence data by using the first enhancement manner, and the third time sequence enhancement data is time sequence data obtained by performing enhancement on the first sub-time sequence data or the second sub-time sequence data by using a second enhancement manner; the first sub-time sequence data and the second sub-time sequence data are two different sub-time sequence data in the first time sequence data, and the first time sequence data is any time sequence data in the plurality of time sequence data.

[0120] In a possible implementation, the training module 802 is further configured to train a visual feature extraction model according to the positive sample pairs and the negative sample pairs, to obtain a trained visual feature extraction model, and the visual feature extraction model is configured to extract a feature vector of image data.

[0121] In a possible implementation, the visual feature extraction model is a model obtained by training a visual base model by using a plurality of image data, and the visual feature extraction model has the ability to extract structural information representation in image data.

[0122] FIG. 9 is a structural schematic diagram of a feature extraction model training apparatus according to an embodiment of the present application. The apparatus is applied to a computer device, and the computer device is configured to execute the method shown in FIG. 6. Based on a plurality of modules shown in FIG. 9, the feature extraction model training apparatus shown in FIG. 9 can execute all or part of operations performed by the computer device. It should be understood that the apparatus can include more additional modules than the shown modules or omit part of the shown modules, and the embodiments of the present application do not limit this. As shown in FIG. 9, the apparatus includes:

[0123] The first obtaining module 901 is configured to obtain time-series data to be processed.

[0124] The extraction module 902 is configured to extract a feature vector of the time-series data based on a feature extraction model, the feature extraction model comprising a time-series feature extraction model and / or a visual feature extraction model, the time-series feature extraction model being trained by the training method shown in FIG. 2, and the visual feature extraction model being trained by the training method shown in FIG. 2.

[0125] The second obtaining module 903 is configured to obtain a processing result of the time-series data according to the feature vector of the time-series data.

[0126] In a possible implementation, the time-series data to be processed belongs to a first task scenario; the device further comprises a fine-tuning module configured to fine-tune the feature extraction model by a plurality of first time-series data in the first task scenario to obtain a fine-tuned feature extraction model; and the extraction module 902 is configured to call the fine-tuned feature extraction model to obtain the feature vector of the time-series data.

[0127] In a possible implementation, the fine-tuning module is configured to obtain a plurality of first sample pairs based on the plurality of first time-series data, the plurality of first sample pairs comprising positive first sample pairs and negative first sample pairs; and fine-tune the feature extraction model according to the positive first sample pairs and the negative first sample pairs to obtain the fine-tuned feature extraction model.

[0128] In a possible implementation, the extraction module 902 is configured to call the feature extraction model to obtain the feature vector of the time-series data.

[0129] In a possible implementation, the second obtaining module 903 is configured to obtain the processing result of the time-series data according to the feature vector of the time-series data and a task network, the task network comprising a classification network, a prediction network or a regression network.

[0130] It should be understood that the device provided in the above embodiments is only used as an example for the division of the functional modules, and in actual applications, the above functions can be completed by different functional modules according to the needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the above described functions. In addition, the device and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process and beneficial effects are described in the method embodiments, which will not be repeated here.

[0131] Referring to FIG. 10, FIG. 10 shows a structural diagram of a computer device 2000 according to an example embodiment of the present application. The computer device 2000 shown in FIG. 10 is configured to perform operations involved in the training method of the feature extraction model shown in FIG. 2 or the processing method of time series data shown in FIG. 6. The computer device 2000 can be a computer, a tablet, or the like, and can be implemented by a general bus architecture.

[0132] As shown in FIG. 10, the computer device 2000 includes at least one processor 2001, a memory 2003, and at least one communication interface 2004.

[0133] The processor 2001 can be a general central processing unit (CPU), a digital signal processor (DSP), a network processer (NP), a graphics processing unit (GPU), a neural-network processing units (NPU), a data processing unit (DPU), a microprocessor, or one or more integrated circuits for implementing the schemes of the present application. For example, the processor 2001 can include an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or other programmable logic device, transistor logic, a hardware component, or any combination thereof. The PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. The processor can implement or execute various logical blocks, modules, and circuits described in connection with the disclosure of the embodiments of the present application. The processor can also be a combination of computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0134] Optionally, the computer device 2000 further includes a bus. The bus is used to transmit information between the components of the computer device 2000. The bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is used in FIG. 10, but it does not mean that there is only one bus or only one type of bus.

[0135] The memory 2003 is, for example, a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM), or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, a magnetic disk storage or other magnetic storage devices, or any other medium capable of storing desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto. The memory 2003 is, for example, independent and connected to the processor 2001 through the bus. The memory 2003 can also be integrated with the processor 2001.

[0136] The communication interface 2004 uses any transceiver-like mechanism for communicating with other devices or a communication network, which can be an Ethernet network, a radio access network (RAN), a wireless local area networks (WLAN), etc. The communication interface 2004 can include a wired communication interface and / or a wireless communication interface. Specifically, the communication interface 2004 can be an Ethernet interface, a Fast Ethernet (FE) interface, a Gigabit Ethernet (GE) interface, an Asynchronous Transfer Mode (ATM) interface, a wireless local area networks (WLAN) interface, a cellular network communication interface, or a combination thereof. The Ethernet interface can be an optical interface, an electrical interface, or a combination thereof. In embodiments of the present application, the communication interface 2004 can be used for the computer device 2000 to communicate with other devices.

[0137] In particular implementations, as one example, the processor 2001 can include one or more CPUs, such as CPU0 and CPU1 as shown in FIG. 10. Each of these processors can be a single-core (single-core CPU) processor or a multi-core (multi-core CPU) processor. A processor, as used herein, can refer to one or more devices, circuits, and / or processing cores for processing data, such as computer program instructions.

[0138] In particular implementations, as one example, the computer device 2000 can include multiple processors, such as the processor 2001 and the processor 2005 as shown in FIG. 10. Each of these processors can be a single-core (single-core CPU) processor or a multi-core (multi-core CPU) processor. A processor, as used herein, can refer to one or more devices, circuits, and / or processing cores for processing data, such as computer program instructions.

[0139] In specific implementations, as an embodiment, the computer device 2000 can further include an output device and an input device. The output device and the processor 2001 communicate, and can display information in various ways. For example, the output device can be a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector, etc. The input device and the processor 2001 communicate, and can receive user input in various ways. For example, the input device can be a mouse, a keyboard, a touch screen device, or a sensor device, etc.

[0140] In some embodiments, the memory 2003 is configured to store program code 2010 for implementing the solutions of the present application, and the processor 2001 can execute the program code 2010 stored in the memory 2003. That is, the computer device 2000 can implement the training method of the feature extraction model or the processing method of the time series data provided by the method embodiments through the processor 2001 and the program code 2010 in the memory 2003. The program code 2010 can include one or more software modules. Alternatively, the processor 2001 itself can also store program codes or instructions for implementing the solutions of the present application.

[0141] In specific embodiments, the computer device 2000 of the embodiments of the present application can correspond to the computer device in the above-mentioned various method embodiments. The processor 2001 in the computer device 2000 reads instructions in the memory 2003, so that the computer device 2000 shown in FIG. 10 can perform all or part of the operations performed by the computer device.

[0142] Specifically, the processor 2001 is configured to obtain a plurality of time series data, the plurality of time series data being derived from a plurality of task scenarios; obtain a plurality of sample pairs based on the plurality of time series data, the plurality of sample pairs including positive sample pairs and negative sample pairs, the positive sample pairs including first time series data and first image data, and the negative sample pairs including the first time series data and second image data; the first time series data being any time series data in the plurality of time series data, the first image data being image data obtained by modal conversion on the first time series data, and the second image data being image data obtained by modal conversion on second time series data, the second time series data being time series data other than the first time series data in the plurality of time series data; train a time series feature extraction model according to the positive sample pairs and the negative sample pairs, to obtain a trained time series feature extraction model, and the time series feature extraction model being configured to extract a feature vector of the time series data.

[0143] Alternatively, the processor 2001 is configured to acquire time-series data to be processed; extract a feature vector of the time-series data based on a feature extraction model, the feature extraction model comprising a time-series feature extraction model and / or a visual feature extraction model, the time-series feature extraction model being trained by the training method shown in FIG. 2, and the visual feature extraction model being trained by the training method shown in FIG. 2; and acquire a processing result of the time-series data according to the feature vector of the time-series data.

[0144] Other optional embodiments will not be described herein for brevity.

[0145] The computer device 2000 can also correspond to the feature extraction model training apparatus shown in FIG. 8 or the time-series data processing apparatus shown in FIG. 9. Each functional module in the apparatus is implemented by software of the computer device 2000. In other words, the functional modules included in the feature extraction model training apparatus or the time-series data processing apparatus are generated by the processor 2001 of the computer device 2000 after reading the program code 2010 stored in the memory 2003.

[0146] The steps of the method shown in FIG. 2 or FIG. 6 are completed by the integrated logic circuit of hardware or the instructions in the form of software in the processor of the computer device 2000. The steps of the method disclosed in the embodiments of the present application can be directly embodied as being completed by a hardware processor, or being completed by a combination of hardware and software modules in the processor. The software module can be located in a storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, or the like. The storage medium is located in the memory, and the processor reads information in the memory and combines hardware to complete the steps of the above method. To avoid repetition, the above will not be described in detail.

[0147] FIG. 11 is a structural schematic diagram of a server provided in an embodiment of the present application. The server 1000 can be quite different in configuration or performance, and can include one or more processors 1001 and one or more memories 1002, wherein the one or more memories 1002 store at least one computer program, the at least one computer program is loaded and executed by the one or more processors 1001, so that the server implements the method provided in each of the above method embodiments. Of course, the server 1000 can also have a wired or wireless network interface, a keyboard, an input and output interface, and the like, so as to perform input and output. The server 1000 can also include other components for realizing the functions of the device, which will not be described herein.

[0148] It is to be understood that the above-described processor can be a CPU, and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, and the like. The general-purpose processor can be a microprocessor or any conventional processor, and the like. It is to be noted that the processor can be an advanced RISC machines (ARM) architecture processor.

[0149] Further, in an optional embodiment, the above-described memory can include a read-only memory and a random access memory, and provide instructions and data to the processor. The memory can also include a non-volatile random access memory. For example, the memory can also store device type information.

[0150] The memory can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM) and direct rambus RAM (DR RAM).

[0151] The embodiment of the present application further provides a computer readable storage medium, at least one instruction is stored in the storage medium, the instruction is loaded and executed by a processor, so that the computer implements the training method of the feature extraction model or the processing method of the time series data according to any one of the above.

[0152] The embodiment of the present application further provides a computer program (product), when the computer program is executed by a computer, the processor or the computer can execute the corresponding steps and / or processes in the above method embodiments.

[0153] The embodiment of the present application further provides a chip, comprising a processor, for calling and running instructions stored in a memory, so that a communication device installed with the chip executes the training method of the feature extraction model or the processing method of the time series data according to any one of the above.

[0154] The embodiment of the present application further provides another chip, comprising: an input interface, an output interface, a processor and a memory, the input interface, the output interface, the processor and the memory are connected through internal connection paths, the processor is used to execute the code in the memory, when the code is executed, the processor is used to execute the training method of the feature extraction model or the processing method of the time series data according to any one of the above.

[0155] In the above embodiments, all or part of them can be realized by software, hardware, firmware or any combination thereof. When realized by software, all or part of them can be realized in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be magnetic media (such as floppy disk, hard disk, magnetic tape), optical media (such as DVD) or semiconductor media (such as solid state disk) and the like.

[0156] Those skilled in the art can appreciate that, in combination with the method steps and modules described in the embodiments disclosed herein, all or part of the steps can be implemented by software, hardware, firmware or any combination thereof. In order to clearly illustrate the interchangeability of hardware and software, the steps and components of the embodiments have been described in the above description in general terms. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0157] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or by programs instructing related hardware, which can be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0158] When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer program instructions. As an example, the method of the embodiments of the present application can be described in the context of machine-executable instructions, such as program modules that are executed by devices included in the target real or virtual processor. Generally, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., which perform specific tasks or implement specific abstract data structures. In various embodiments, the functions of the program modules can be combined or divided among the described program modules. Machine-executable instructions for program modules can be executed within a local or distributed device. In a distributed device, program modules can be located in both local and remote storage media.

[0159] The computer program code for implementing the method of the embodiments of the present application can be written in one or more programming languages. These computer program codes can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing apparatus, so that when the computer program codes are executed by the computer or other programmable data processing apparatus, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program codes can be executed entirely on the computer, partially on the computer, as a separate software package, partially on the computer and partially on a remote computer, or entirely on a remote computer or server.

[0160] In the context of the embodiments of the present application, computer program codes or related data can be carried by any appropriate carrier to enable the device, apparatus or processor to perform the various processes and operations described above. Examples of the carrier include a signal, a computer readable medium, etc.

[0161] Examples of a signal can include electrical, optical, radio frequency, sound, or other forms of propagated signals, such as carrier waves, infrared signals, etc.

[0162] A machine-readable medium can be any tangible medium that contains or stores the program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), and a digital versatile disc (DVD), or any suitable combination of the foregoing.

[0163] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device and module can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0164] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are merely illustrative, for example, the division of the module is only a logical function division, and actual implementation can have another division manner, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be indirect coupling or communication connection through some interfaces, devices or modules, and can also be electrical, mechanical or other form of connection.

[0165] The module described as a separate component can or can not be physically separated, and the component displayed as a module can or can not be a physical module, that is, it can be located in one place, or can be distributed to a plurality of network modules. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0166] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically, or two or more modules can be integrated into one module. The above integrated module can be realized in the form of hardware or in the form of software functional module.

[0167] The integrated module, if implemented in the form of a software function module and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art, or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0168] The terms "first", "second", and the like in the present application are used to distinguish between items or similar items having substantially the same function and action. It should be understood that there is no logical or chronological dependency between "first", "second", and "nth", and the number and execution order are not limited. It should also be understood that although the following description uses the terms first, second, and the like to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of various examples, a first image can be referred to as a second image, and similarly, a second image can be referred to as a first image. The first image and the second image can both be images, and in some cases, can be separate and distinct images.

[0169] It should also be understood that in various embodiments of the present application, the size of the serial number of each process does not mean the order of execution, and the execution order of each process should be determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0170] In the present application, the term "at least one" means one or more, and the term "multiple" in the present application means two or more, for example, multiple second messages refer to two or more second messages. The terms "system" and "network" are often used interchangeably in this document.

[0171] It should be understood that the terms used in the description of various described examples herein are only for the purpose of describing specific examples and are not intended to be limiting. As used in the description of various described examples and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0172] It should also be understood that, as used in this specification, the terms "comprises", "comprising", "includes", "including", "with" or "comprising", specifies the presence of stated features, integers, steps, operations, elements, and / or components but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0173] It should also be understood that the terms "comprises", "comprising", "includes", "including", "with" or "comprising", specifies the presence of stated features, integers, steps, operations, elements, and / or components but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0174] It should also be understood that the terms "if' and "when" can be construed to mean "upon" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [a stated condition or event] is detected" can be construed to mean "upon determining" or "in response to determining" or "upon detecting [the stated condition or event]" or "in response to detecting [the stated condition or event]", depending on the context.

[0175] It should be understood that a determination of B according to A does not mean that B is determined only according to A, but B can also be determined according to A and / or other information.

[0176] It should also be understood that the terms "one embodiment", "an embodiment", "one possible implementation", as used in the specification are intended to mean that a particular feature, structure, or characteristic described in connection with the embodiment or implementation is included in at least one implementation of the application. Thus, appearances of the phrases "in one embodiment" or "in an embodiment", "one possible implementation", as well as variations thereof, in various places throughout the specification are not necessarily intended to refer to the same embodiment or implementation, unless otherwise indicated. Furthermore, it is appreciated that the various particular features, structures, or characteristics described in connection with any of the embodiments or implementations can be combined in any suitable manner in one or more implementations.

[0177] The above description is only optional embodiments of the application, and is not used to limit the application, any modification, equivalent replacement, improvement, etc. made within the principles of the application should be included in the protection scope of the application.

Claims

1. A method for training a feature extraction model, characterized in that, The method comprises: obtaining a plurality of time series data, the plurality of time series data being derived from a plurality of task scenarios; obtaining a plurality of sample pairs based on the plurality of time series data, the plurality of sample pairs comprising positive sample pairs and negative sample pairs, the positive sample pairs comprising first time series data and first image data, and the negative sample pairs comprising the first time series data and second image data; the first time series data being any time series data in the plurality of time series data, the first image data being image data obtained by modal conversion of the first time series data, and the second image data being image data obtained by modal conversion of second time series data, the second time series data being time series data other than the first time series data in the plurality of time series data; training a time series feature extraction model according to the positive sample pairs and the negative sample pairs to obtain a trained time series feature extraction model, the time series feature extraction model being configured to extract a feature vector of time series data.

2. The method of claim 1, wherein, The positive sample pairs further comprise first time series enhancement data and second time series enhancement data, and the negative sample pairs further comprise the first time series enhancement data and third time series enhancement data; the first time series enhancement data being time series data obtained by enhancing first sub-time series data using a first enhancement manner, the second time series enhancement data being time series data obtained by enhancing second sub-time series data using the first enhancement manner, and the third time series enhancement data being time series data obtained by enhancing the first sub-time series data or the second sub-time series data using a second enhancement manner; the first sub-time series data and the second sub-time series data being two different sub-time series data in the first time series data, and the first time series data being any time series data in the plurality of time series data.

3. The method according to claim 1 or 2, characterized in that, The method further comprises: training a visual feature extraction model according to the positive sample pairs and the negative sample pairs to obtain a trained visual feature extraction model, the visual feature extraction model being configured to extract a feature vector of image data.

4. The method of claim 3, wherein, The visual feature extraction model is a model obtained by training a visual base model using a plurality of image data, and the visual feature extraction model has the ability to extract structural information representation in image data.

5. A method of processing time series data, the method comprising: The method comprises: obtaining time series data to be processed; extracting a feature vector of the time series data based on a feature extraction model, the feature extraction model comprising a time series feature extraction model and / or a visual feature extraction model, the time series feature extraction model being a time series feature extraction model trained by the training method of claim 1 or 2, and the visual feature extraction model being a visual feature extraction model trained by the training method of claim 3 or 4; obtaining a processing result of the time series data according to the feature vector of the time series data.

6. The method of claim 5, wherein, The time series data to be processed belongs to a first task scenario; and the extracting of the feature vector of the time series data based on the feature extraction model comprises: fine-tuning the feature extraction model using a plurality of first time series data in the first task scenario to obtain a fine-tuned feature extraction model; and obtaining the feature vector of the time series data by invoking the fine-tuned feature extraction model.

7. The method of claim 6, wherein, The fine-tuning of the feature extraction model through the plurality of first time sequence data under the first task scene comprises: A plurality of first sample pairs are obtained based on the plurality of first time sequence data, and the plurality of first sample pairs comprise positive first sample pairs and negative first sample pairs; The feature extraction model is fine-tuned according to the positive first sample pairs and the negative first sample pairs, and a fine-tuned feature extraction model is obtained.

8. The method of claim 5, wherein, The feature vector of the time sequence data is extracted based on the feature extraction model, which comprises: The feature vector of the time sequence data is obtained by calling the feature extraction model.

9. The method according to any of claims 5-8, characterized by, The processing result of the time sequence data is obtained according to the feature vector of the time sequence data, which comprises: The processing result of the time sequence data is obtained according to the feature vector of the time sequence data and a task network, and the task network comprises a classification network, a prediction network or a regression network.

10. An apparatus for training a feature extraction model, comprising: The device comprises: An acquisition module is configured to acquire a plurality of time sequence data, and the plurality of time sequence data is derived from a plurality of task scenes; The acquisition module is further configured to acquire a plurality of sample pairs based on the plurality of time sequence data, and the plurality of sample pairs comprise positive sample pairs and negative sample pairs, the positive sample pairs comprise first time sequence data and first image data, and the negative sample pairs comprise the first time sequence data and second image data; the first time sequence data is any time sequence data in the plurality of time sequence data, the first image data is image data obtained by modal conversion of the first time sequence data, and the second image data is image data obtained by modal conversion of second time sequence data, which is time sequence data other than the first time sequence data in the plurality of time sequence data; A training module is configured to train a time sequence feature extraction model according to the positive sample pairs and the negative sample pairs, and obtain a trained time sequence feature extraction model, wherein the time sequence feature extraction model is used to extract a feature vector of time sequence data.

11. A processing device of time series data, characterized by, The device comprises: A first acquisition module is configured to acquire time sequence data to be processed; An extraction module is configured to extract a feature vector of the time sequence data based on a feature extraction model, wherein the feature extraction model comprises a time sequence feature extraction model and / or a visual feature extraction model, the time sequence feature extraction model is a time sequence feature extraction model trained by the training method of claim 1 or 2, and the visual feature extraction model is a visual feature extraction model trained by the training method of claim 3 or 4; A second acquisition module is configured to obtain a processing result of the time sequence data according to the feature vector of the time sequence data.

12. A computer device, comprising: The computer device comprises a processor and a memory, the memory stores at least one program instruction or code, the at least one program instruction or code is loaded and executed by the processor, so that the computer device implements the method of any one of claims 1-9.

13. A computer-readable storage medium, characterized in that, The computer storage medium stores at least one instruction, the at least one instruction is loaded and executed by the processor, so that the computer device implements the method of any one of claims 1-9.

14. A computer program product, characterised in that, The computer program product comprises computer program code which is loaded and executed by a computer to cause the computer device to implement the method of any one of claims 1-9.

Images