Multi-modal data acquisition and processing method based on reflection mechanism

Abstract

The invention is applicable to the technical field of data acquisition, and provides a multi-modal data acquisition and processing method based on a reflection mechanism, which comprises the following steps: a client encapsulates request information and sends the request information to a server; the server side verifies and processes request parameters in the request information; the server automatically identifies the data source type and version of the multi-modal data, dynamically finds the reflected class and the method and attribute of the class by using a reflection mechanism, and obtains corresponding data element information; and analyzing and processing the extracted structured, semi-structured and non-structured data in a corresponding mode, finally performing feature fusion, and returning a result to the client after the feature fusion is completed. According to the method, a multi-modal data processing framework is constructed, a powerful data preprocessing and feature extraction process and a feature fusion method based on machine learning are designed, the problems of multi-modal data acquisition and processing are efficiently and uniformly solved, and efficient analysis and intelligent decision can be promoted.

Description

Technical Field

[0001] The present invention belongs to the technical field of data acquisition and data analysis, and in particular relates to a multimodal data acquisition and processing method based on a reflection mechanism. Background Art

[0002] With the rapid development of information technology, the types and forms of data are becoming increasingly diverse, expanding from traditional structured data, text and pictures to multimodal data such as video and audio. These data usually come from different platforms and devices, with different formats and standards. How to efficiently obtain these multimodal data from different sources and formats has become a research focus. Existing data acquisition methods often require writing code for each data source to adapt to different data formats and versions. This method is not only inefficient but also difficult to maintain.

[0003] To address the above challenges, researchers have developed various technologies and methods, such as the multimodal learning framework in deep learning, which can automatically learn useful features from data of different modalities and try to fuse these features to improve task performance. In addition, there are algorithms and technologies designed specifically for specific problems, such as cross-modal retrieval systems for solving the semantic gap, or dimensionality reduction techniques designed to process high-dimensional data.

[0004] However, current methods still cannot efficiently and uniformly solve the problem of multimodal data acquisition and processing. Summary of the invention

[0005] In view of the above problems, the purpose of the present invention is to provide a multimodal data acquisition and processing method based on a reflection mechanism, aiming to solve the technical problem.

[0006] The present invention adopts the following technical solution:

[0007] The multimodal data acquisition and processing method based on the reflection mechanism comprises the following steps:

[0008] Step S1, the client encapsulates the request information and sends it to the server;

[0009] Step S2: The server receives the request information from the client, verifies and processes the request parameters in the request information;

[0010] Step S3: According to the request parameters, the server automatically identifies the data source type and version of the multimodal data, uses the reflection mechanism to dynamically find the reflected class and the class methods and properties, and obtains the corresponding data metadata;

[0011] Step S4: extracting structured data from the data source and processing the structured data;

[0012] Step S5: extracting semi-structured data from the data source and processing the semi-structured data;

[0013] Step S6: for unstructured data, including unstructured text data and image data, data collection, cleaning, conversion, storage and analysis are performed to extract useful information and knowledge;

[0014] Step S7: Fusing the extracted data features of various types of data to obtain the overall features of the input text.

[0015] The beneficial effects of the present invention are as follows: the present invention forms a complete data processing process, and for multimodal data with diverse sources and complex structures, various types of data can be flexibly processed through a unified data processing framework; at the same time, for compatibility issues that may arise during data integration, the present invention solves the compatibility issues of multimodal data integration through data conversion. In short, through the flexible data acquisition and processing method of the present invention, efficient analysis and intelligent decision-making of multimodal data are achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of a multimodal data acquisition and processing method based on a reflection mechanism provided by an embodiment of the present invention;

[0017] Figure 2 It is a processing flow chart of structured data;

[0018] Figure 3 It is a processing flow chart for semi-structured data;

[0019] Figure 4 It is a processing flow chart of unstructured text data;

[0020] Figure 5 It is a processing flow chart of unstructured image data;

[0021] Figure 6 It is a specific flow chart of step S7. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0023] In order to illustrate the technical solution of the present invention, a specific embodiment is provided below for illustration.

[0024] like Figure 1 As shown, the multimodal data acquisition and processing method based on the reflection mechanism provided in this embodiment includes the following steps:

[0025] Step S1: The client encapsulates the request information and sends it to the server.

[0026] For request messages, the client has a fixed encapsulation format. In this embodiment, the request information is encapsulated in JSON format and sent to the server. Specifically, the process of this step is as follows:

[0027] S11. Determine the request parameters and construct a JSON object, which includes all necessary key-value pairs. Then, the client encapsulates the request information in JSON format as required. The request parameters of the request information include the parameter name and parameter value corresponding to the parameter.

[0028] As a specific example, the parameter name of the request parameter includes the name of the data source dataSourceName, the IP address of the data source dataSourceIP, the time type timeType, the creation time createTime, and the catalog ID catalogId. The catalog ID is used to specify a specific catalog or database, otherwise it can be left blank.

[0029] For example, a specific request parameter in JSON format:

[0030] {"datasourceName":"datasource172","datasourceName":"datasource172","datasourceIP":"172.18.8.112","timeType":

[0031] "createTime","catalogId":""}.

[0032] S12. Use the JSON object as the request body and send the request to the corresponding API endpoint of the server using the HTTP POST method.

[0033] According to the determined request parameters, a JSON object is constructed, containing all necessary key-value pairs. The JSON object is used as the request body and the HTTP request is sent using the HTTP POST method. The client (such as curl, Postman, HTTP library in programming language, etc.) sends the above JSON object to the corresponding API endpoint of the server. The URL of the server endpoint needs to be determined according to the actual service address.

[0034] S13. When receiving the HTTP response returned by the server, the client parses the response.

[0035] After sending the HTTP request to the server, the server returns an HTTP response after processing. The HTTP response is usually in JSON format and contains information about the library and table. See the following description for details.

[0036] Step S2: The server receives the request information from the client, verifies and processes the request parameters in the request information;

[0037] The server responds to the request information, and after necessary verification and query processing, it will give the client a specific feedback indicating the query result, indicating that the server has matched the corresponding library and table according to the request information. In the subsequent steps, the multimodal data will be further processed according to the quantity type, and finally integrated and returned to the client as a whole. Specifically, the process of returning the HTTP response in this step is as follows:

[0038] S21. The server listens to the client's request information through the API endpoint and receives the request information through the Web server framework.

[0039] The server needs to have an API endpoint (such as / api / getDataSources) to listen to the client's request information and use a web server framework (such as Express.js, Django, Flask, etc.) to receive HTTP requests.

[0040] S22. Use the library to parse the request information.

[0041] Use the corresponding library, such as body-parser, to parse the JSON request parameter data in the request information.

[0042] S23. Check whether the request parameters are complete.

[0043] Verify whether the request parameters are complete, for example, whether key request parameters such as dataSourceName and dataSourceIP exist.

[0044] S24. When the verification is successful, the corresponding business logic is obtained according to the request parameters.

[0045] For example, the business logic is to query the database to obtain the library and table information under the directory corresponding to the catalogId under the data source.

[0046] S25. Connect to the database and execute SQL query to obtain the library and table information under the data source, and convert the query results into JSON format for use by the client.

[0047] S26. Encapsulate the query result data into the response body, set the response status code, and construct an HTTP response. For example, the code "200OK" indicates success.

[0048] S27. When the verification fails or the query fails, the corresponding error message and status code are returned, such as "400BadRequest" indicating that the request parameters are incorrect.

[0049] S28. Send the constructed HTTP response to the client.

[0050] Step S3: Based on the request parameters, the server automatically identifies the data source type and version of the multimodal data, uses the reflection mechanism to dynamically find the reflected class and the class methods and properties, and obtains the corresponding data metadata. The specific process is as follows:

[0051] S31, selecting at least one data source from the multimodal data as a target data source, and determining a corresponding adapter according to the type and version of the target data source;

[0052] S32. Use the reflection mechanism to obtain the corresponding data element information:

[0053] Get the Class object of the data source class through the Class.forName() method, and then use the getFields() or getMethods() method to obtain the public fields and method information of the class to construct the data metadata information of the data source.

[0054] Once the data source type and version are identified, the server can use the reflection mechanism to dynamically obtain the metadata of the data source, specifically through the getFields() or getMethods() method, etc., and then use it for multimodal data processing and analysis.

[0055] Step S4: extracting structured data from the data source and processing the structured data;

[0056] Extract structured data from multiple data sources, such as database tables. For example, collect structured data from MySQL, DAMO database, Renmin University of China Golden Warehouse, etc. Figure 2 As shown, the processing of structured data is as follows:

[0057] S41. Extract structured data from the data source.

[0058] S42. Data preprocessing mainly involves cleaning and preparing the raw data, including processing missing values, removing outliers, and formatting data. The mean filling algorithm is used to replace missing values ​​with the mean of the feature, and the interquartile range (IQR) is used to identify outliers. IQR is the difference between the third quartile (Q3) and the first quartile (Q1). Data points that are usually lower than Q1-1.5*IQR or higher than Q3+1.5*IQR are considered outliers.

[0059] S43. Perform feature conversion on the data, specifically using the distance measurement algorithm K nearest neighbor algorithm, and ensure that all features are at a similar order of magnitude.

[0060] S44. Encode the data: Use one_hot encoding to convert a single feature into a multi-binary feature, which helps to improve the expressiveness of the model. The tabular data can be converted into an m×t matrix, where m is the sample size and t is the number of feature encoding bits.

[0061] For example: gender (male 1, female 2, occupies one position), age (0-999, occupies three positions), operation type (1 export, 2 import, 3 click to view, occupies one position), as shown in the following table:

[0062] Id gender age Operation Type 1 female 30 Export 2 male 31 Import 3 female 29 Click to view 4 male 32 Import

[0063] The corresponding encoding is: {1, 0, 3, 0, 1}, {0, 0, 3, 1, 2}, {1, 0, 2, 9, 3}, {0, 0, 3, 2, 2}, which is converted into a 4×5 feature matrix.

[0064] S45. The encoded data is expanded into a three-dimensional matrix of m×t×1.

[0065] Step S5: extracting semi-structured data from the data source and processing the semi-structured data;

[0066] The sources of semi-structured data include Web data, log files, JSON and XML files, including text, images, videos, etc. Taking JSON files as an example, the specific process of step S5 is as follows:

[0067] S51. Extract semi-structured data from the data source.

[0068] S52. Data preprocessing includes data cleaning, format conversion, and data separation. Data cleaning uses the data processing library Pandas to remove useless symbols, spaces, and special characters, and to process missing values ​​and outliers. Format conversion is used to convert data into a unified format.

[0069] For example, convert a date string to a date object and parse a JSON string into a dictionary, such as Json_data = {"name":"ty","age":22,"skills":["Java","C++","Python"],"address":{"city":"Shanghai","district":"Pudong"}}

[0070] Converted to: {"name":"ty","age":22,"skills":["Java","C++","Python"],"address_city":"Shanghai","address_district":"Pudong"}。

[0071] S53. Encode the data: Use Unicode encoding. Variants such as UTF-8 and UTF-16 allow these characters to be stored efficiently in different ways. Extract each feature separately, similar to the encoding of tabular data, and convert the data into a k×w matrix, where k is the number of samples and w is the number of bits for feature encoding.

[0072] S54. Expand the encoded data into a three-dimensional matrix of k×w×1.

[0073] Step S6. For unstructured data, including unstructured text data and image data, perform data collection, cleaning, transformation, storage, and analysis to extract useful information and knowledge;

[0074] This step is divided into the processing of unstructured text data and the processing of unstructured image data.

[0075] For unstructured text data, such as Figure 3 As shown, the data processing process is as follows:

[0076] S611. Obtain text data: Collect or generate the text data to be processed from existing corpora.

[0077] S612. Tokenize: For Chinese text, use jieba to perform tokenization. For Chinese text, since there is no clear word boundary, jieba can effectively segment Chinese sentences and supports custom vocabulary to improve the tokenization accuracy of text in specific domains.

[0078] S613. Clean: Use the regular expression library in Python or the BeautifulSoup tool to remove irrelevant information in the text, such as HTML tags, special symbols, etc.

[0079] S614. Standardize: Unify the text format, convert all letters to lowercase, remove punctuation marks, perform stemming or lemmatization to reduce lexical variants.

[0080] S615. Remove stop words: Filter out stop words by loading a predefined stop word list. Stop words refer to those common words that are not contributive in text analysis, such as "is", "of", etc. This can be achieved by loading a predefined stop word list and filtering out these words.

[0081] S616: Input a text T of length l, and segment the text T. The segmentation granularity is character level, that is, each character will be regarded as an independent token.

[0082] The formula is: T = {w1,w2,L,w l},w i Represents the i-th character, 1≤i≤l.

[0083] S617, each Chinese character w i Corresponding to a word vector c i .

[0084] In BERT, the hidden layer dimension defaults to 768 dimensions, so each character will be converted into a 768-dimensional vector representation, that is, c i =[a1,a2,L,a 768 ]. Therefore, each character w i With the word vector c i The corresponding relationship is: i →c i =[a1,a2,L,a 768 ].

[0085] S618. The word vector representation of each character corresponds to a row in the embedding matrix. The character representation vector matrix of the input text is: C = [c1, c2, L, c l ]. The size of matrix C is l×768, c i is the word vector for the ith character.

[0086] S619: Extract glyph features from the text and convert the glyphs into a glyph vector matrix.

[0087] After the above processing, each document can be represented as a row of vectors, and the entire corpus forms a two-dimensional matrix. In this matrix, each row corresponds to a document, and each column represents a feature (i.e., a word).

[0088] For unstructured image data, such as Figure 4 As shown, the data processing process is as follows:

[0089] S621, image preprocessing: image preprocessing is used to improve image quality, simplify data and enhance the detectability of relevant information. The specific steps are as follows:

[0090] (1) Grayscale the image

[0091] Convert color images to grayscale images to reduce the amount of data and simplify the processing. The weighted average method is used here:

[0092] Y=0.299R+0.587G+0.114B.

[0093] (2) Scaling, rotating, and translating the image to correct the geometric distortion of the image

[0094] (3) Image enhancement

[0095] This step enhances the useful information in the enhanced image, and the enhancement process is as follows;

[0096] (31) Performing color adjustment on the image to complete the chromatic aberration correction of the image and obtain a spatial domain reconstructed image;

[0097]

[0098] (32) Dynamically expand the corrected image

[0099] R(x)=round[127.5+ω*R c (p)]

[0100] (33)Finally, normalization

[0101]

[0102] S622, Feature Extraction:

[0103] (1) First, the Gamma correction method is used to standardize the color space of the input image, adjust the contrast of the image, reduce the impact of local shadows and lighting changes in the image, and suppress noise interference;

[0104] (2) Then calculate the gradient (including size and direction) of each pixel in the image; this is mainly to capture contour information while further weakening the interference of lighting;

[0105] (3) Each cell then constructs a gradient direction histogram to provide a code for the local image region while maintaining weak sensitivity to the posture and appearance of human objects in the image;

[0106] (4) Finally, the cell units are combined into large blocks, the gradient histogram is normalized within the block, and the feature vectors of all cells in a block are concatenated to obtain the HOG feature of the block.

[0107] S623, collecting HOG features: collecting HOG features of all overlapping blocks in the detection window, and combining them into a final feature vector for classification.

[0108] Step S7: Fusing the extracted data features of various types of data to obtain the overall features of the input text.

[0109] This step combines the features of the extracted data of various types (such as characters, glyphs, pinyin, and parts of speech) to obtain the overall features of the input text. Figure 6 As shown, the specific process is as follows:

[0110] S71. Normalize the features of the processed structured data, semi-structured data, unstructured text data and image data respectively.

[0111] Normalization is used to ensure that the numerical range of the data is similar, so as to avoid certain features dominating the result of the entire dot product due to their large numerical range.

[0112] S72. Use embedding technology to map glyph, pinyin and part-of-speech features into a high-dimensional space and align them with character features.

[0113] Character features are already high-dimensional features and remain unchanged at a size of N×768 dimensions; glyph features are mapped to N×768 dimensions through embedding technology; pinyin features are mapped to N×768 dimensions through embedding technology; and part-of-speech features are mapped to N×768 dimensions through embedding technology.

[0114] S73. After all features are adjusted to the same dimension, they are concatenated into a feature matrix.

[0115] The columns are concatenated into a large feature matrix as the input matrix X, with a size of N×(3×768)=N×2304 dimensions, combining different feature vectors to enrich the description of a single sample.

[0116] S74. Input the feature matrix into the scaled dot product attention model for feature fusion. The model uses multiple parallel attention layers, which are performed independently in multiple parallel heads. Each head processes a different subspace to obtain multiple attention heads. The attention heads are spliced ​​together by column. The processing process of each attention head is as follows:

[0117] S741, the input feature matrix is ​​converted to Q through the linear layer u , K u , V u ∈R c×d , are the query matrix, key matrix and value matrix of the u-th attention head respectively, and are calculated as follows:

[0118]

[0119] in, is the corresponding parameter matrix, u represents the uth attention head (1≤u≤8), l represents the key matrix dimension; Q u , K u , V uRepresent the query matrix, key matrix and value matrix of the uth attention head respectively. The weight distribution of value is determined by the similarity between query and key.

[0120] S742, calculate Q by dot product u and K u The similarity.

[0121]

[0122] To scale the factor, ensure that the Score value is within a suitable range, make it smoother and easier to optimize, and avoid the gradient vanishing or exploding problem caused by the large length of the key vector, thereby ensuring the stability of the training.

[0123] S743, use the softmax function to convert these dot products into probability distribution, that is, Q u K u The attention distribution is finally weighted for the matrix V u The sum of is calculated as follows:

[0124] Attention(Q u ,K u ,V u )=softmax(Score(Q u ,K u ))V u

[0125] The softmax function ensures that the sum of the weighted coefficients is 1, achieving a normalized distribution of attention.

[0126] S744. Finally, weighted summation is performed to obtain output.

[0127] The above S741-S744 are the execution operations of one attention head. This embodiment uses 8 parallel attention layers, which are performed independently in multiple parallel heads. Each head processes a different subspace, and 8 attention heads are obtained. Then, the 8 attention heads are spliced ​​together in columns to increase the expression ability of the model. The calculation formula is as follows:

[0128] head u =Attention(Q u ,K u ,V u )

[0129] MultiHead(Q,K,V)=Concat(head1,head2,L,head8)W O

[0130] Repeat the similar operation 8 times to get 8 attention matrix heads u ∈R c×d . Concatenate the 8 attention heads together by column to get the concatenation matrix Concat(head1,head2,L,head8)∈R c×8d In order to make the output have the same structure as the input, multiply it by a weight matrix W O ∈R 8c×3072 Get the final eigenvector matrix MultiHead(Q,K,V)∈R c×3074 .

[0131] S74: Return the final data processing result to the client.

[0132] In summary, the present invention constructs a multimodal data processing framework, which uniformly processes data from different sources and formats (such as images, texts, etc.), so that it can automatically identify and adapt to various data types, designs a powerful data preprocessing and feature extraction process, and solves the compatibility and scalability problems of traditional methods in multimodal data processing; at the same time, the present invention also designs a feature fusion method based on machine learning, which can use the correlation between data to obtain a deeper data representation, and realize efficient analysis and intelligent decision-making of multimodal data.

[0133] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multimodal data acquisition and processing method based on reflection mechanism, characterized in that: The multimodal data acquisition and processing method comprises the following steps: Step S1, the client encapsulates the request information and sends it to the server; Step S2: The server receives the request information from the client, verifies and processes the request parameters in the request information; Step S3: According to the request parameters, the server automatically identifies the data source type and version of the multimodal data, uses the reflection mechanism to dynamically find the reflected class and the class methods and properties, and obtains the corresponding data metadata; Step S4: extracting structured data from the data source and processing the structured data; Step S5: extracting semi-structured data from the data source and processing the semi-structured data; Step S6: for unstructured data, including unstructured text data and image data, data collection, cleaning, conversion, storage and analysis are performed to extract useful information and knowledge; Step S7: Fusing the extracted data features of various types of data to obtain the overall features of the input text.

2. The multimodal data acquisition and processing method based on reflection mechanism as claimed in claim 1, characterized in that: The specific process of step S1 is as follows: S11, determine the request parameters and construct a JSON object, which contains all necessary key-value pairs, and then encapsulate the request information in JSON format according to the requirements of the client, where the request parameters of the request information include the parameter name and parameter value corresponding to the parameter; S12, using the JSON object as the request body, and sending the request to the corresponding API endpoint of the server using the HTTP POST method; S13. When receiving the HTTP response returned by the server, the client parses the response.

3. The multimodal data acquisition and processing method based on reflection mechanism as claimed in claim 2, characterized in that: The parameter names of the request parameters include the name of the data source, the IP address of the data source, the time type, the creation time, and the directory ID.

4. The multimodal data acquisition and processing method based on reflection mechanism as claimed in claim 3, characterized in that: The specific process of step S2 is as follows: S21, the server listens to the client's request information through the API endpoint and receives the request information through the Web server framework; S22, using the library to parse the request information; S23, verify whether the request parameters are complete; S24. When the verification is successful, the corresponding business logic is obtained according to the request parameters; S25, connect to the database and execute SQL query, obtain the library and table information under the data source, and convert the query results into JSON format; S26, encapsulate the query result data into the response body, set the response status code, and construct an HTTP response; S27. When verification fails or query fails, corresponding error information and status code are returned; S28. Send the constructed HTTP response to the client.

5. The multimodal data acquisition and processing method based on reflection mechanism as claimed in claim 3, characterized in that: The specific process of step S3 is as follows: S31, selecting at least one data source from the multimodal data as a target data source, and determining a corresponding adapter according to the type and version of the target data source; S32. Use the reflection mechanism to obtain the corresponding data element information: Get the Class object of the data source class through the Class.forName() method, and then use the getFields() or getMethods() method to obtain the public fields and method information of the class to construct the data metadata information of the data source.

6. The multimodal data acquisition and processing method based on reflection mechanism as claimed in claim 5, characterized in that: The specific process of step S4 is as follows: S41, extracting structured data from a data source; S42, preprocessing the data, including processing missing values, removing outliers, and formatting the data, wherein a mean filling algorithm is used to replace missing values ​​with the mean of the feature, and an interquartile range is used to identify outliers; S43, performing feature conversion on the data; S44. Encode the data: Use one-hot encoding to convert a single feature into a multi-binary feature, and convert the table data into an m×t matrix, where m is the sample size and t is the number of feature encoding bits; S45. The encoded data is expanded into a three-dimensional matrix of m×t×1.

7. The multimodal data acquisition and processing method based on reflection mechanism as claimed in claim 6, characterized in that: The specific process of step S5 is as follows: S51, extracting semi-structured data from data sources; S52. Data preprocessing, including data cleaning, format conversion, and data separation; data cleaning uses the data processing library Pandas to remove useless symbols, spaces, and special characters, and process missing values ​​and outliers; format conversion is used to convert data into a unified format; S53, encode the data: use Unicode encoding, extract each feature separately, and convert the data into a k×w matrix, where k is the sample size and w is the number of feature encoding bits; S54. The encoded data is expanded into a three-dimensional matrix of k×w×1.

8. The multimodal data acquisition and processing method based on reflection mechanism as claimed in claim 7, characterized in that: For unstructured text data, the data processing process is as follows: S611, obtaining text data; S612, word segmentation: For Chinese text, use jieba word segmentation to perform word segmentation operations; S613, Cleaning: Use the regular expression library in Python or the BeautifulSoup tool to remove irrelevant information from the text; S614, Standardization: Unify the text format, convert all letters to lowercase, remove punctuation, and perform stemming or lemmatization to reduce vocabulary variations; S615, removing stop words: by loading a predefined stop word list and filtering out stop words; S616: Input a text T of length l, and segment the text T. The segmentation granularity is character level, that is, each character will be regarded as an independent token. S617, each Chinese character w i Corresponding to a word vector c i ; S618. The word vector representation of each character corresponds to a row in the embedding matrix. The character representation vector matrix of the input text is: C = [c1, c2, L, c l ]; S619: Extract glyph features from the text and convert the glyphs into a glyph vector matrix.

9. The multimodal data acquisition and processing method based on reflection mechanism as claimed in claim 8, characterized in that: For unstructured image data, the data processing process is as follows: S621. Image preprocessing First, the image is grayed out, and then the image is scaled, rotated, and translated to correct the geometric distortion of the image. Finally, the image is enhanced. The enhancement process is as follows: the image is color adjusted, the image chromatic aberration is corrected, the spatial domain reconstructed image is obtained, the corrected image is dynamically expanded, and finally normalized. S622, Feature Extraction: First, the Gamma correction method is used to standardize the color space of the input image and adjust the contrast of the image. Then, the gradient of each pixel of the image is calculated. Then, each cell unit constructs a gradient direction histogram to provide a code for the local image area while maintaining weak sensitivity to the posture and appearance of the human object in the image. Finally, the cell units are combined into large blocks, and the gradient histogram is normalized within the block. The feature vectors of all cells in a block are connected in series to obtain the HOG feature of the block. S623, collect HOG features: All overlapping blocks in the detection window are collected for HOG features and combined into the final feature vector for classification.

10. The multimodal data acquisition and processing method based on reflection mechanism as claimed in claim 9, characterized in that: The specific process of step S7 is as follows: S71, normalizing the features of the processed structured data, semi-structured data, unstructured text data, and image data respectively; S72. Use embedding technology to map glyph, phonetic and part-of-speech features into a high-dimensional space and align them with character features; S73. After all features are adjusted to the same dimension, they are concatenated into a feature matrix: S74. Input the feature matrix into the scaled dot product attention model for feature fusion. The model uses multiple parallel attention layers, which are performed independently in multiple parallel heads. Each head processes a different subspace to obtain multiple attention heads. The attention heads are spliced ​​together by column. The processing process of each attention head is as follows: S741, the input feature matrix is ​​converted to Q through the linear layer u , K u , V u ∈R c×d , are the query matrix, key matrix and value matrix of the u-th attention head respectively; S742, calculate Q by dot product u and K u similarity; S743, use the softmax function to convert these dot products into probability distribution, that is, Q u K u The attention distribution is finally weighted for the matrix V u The sum of S744, finally weighted sum is performed to obtain output; S745. Return the final data processing result to the client.

Images