Unstructured data processing method, apparatus, device, and storage medium

Abstract

The application provides a kind of unstructured data processing method, device, equipment and storage medium, its method includes: obtaining the unstructured data generated in production process;Unstructured data is analyzed by function base, and the characteristic coefficient vector that represents the structural characteristic of unstructured data is obtained;Characteristic coefficient vector is matched with the standard feature library established in advance, and according to the matching result, unstructured data is processed, and data anomaly recognition result is obtained.The application introduces function base analysis, and converts original unstructured data into characteristic coefficient vector with clear mathematical meaning, realizes the unified, compact expression of the internal structure characteristics of data, to quickly and accurately identify whether production data is abnormal, so as to improve production product quality and production management efficiency.

Description

Technical Field

[0001] This application relates to the field of data processing technology in the manufacturing industry, and in particular to a method, apparatus, device and storage medium for processing unstructured data. Background Technology

[0002] With the rapid development of the electronic information industry and computer technology, the scale and complexity of various unstructured data in engineering design, system modeling, and industrial applications are constantly increasing. This data exhibits characteristics such as inconsistent structure, diverse representations, and strong noise interference. Such data often carries key information and functional semantics, and its efficient and accurate parsing and processing directly affects the analysis quality, verification efficiency, and operational reliability of subsequent systems. Automating the parsing, feature extraction, and anomaly detection of unstructured data through computing systems can not only significantly reduce manual processing costs but also improve the consistency, interpretability, and overall system reliability of data processing. This is of great significance for promoting the application of intelligent systems in embedded, low-power, and real-time scenarios.

[0003] During equipment research and development, production, and operation, multi-source unstructured data is generated. For example, during equipment operation monitoring and debugging, sensors, control systems, or detection modules generate time-series signal data. If the time-series signal data has problems such as missing data, abnormal fluctuations, or feature distortion, it indicates that the corresponding components may have aging issues, assembly deviations, electromagnetic interference, or operating condition fluctuations. Similarly, unstructured text data such as equipment design documents, process records, maintenance logs, and quality traceability texts generated during daily office work and equipment operation may have issues such as missing information, inconsistent records, or abnormal operations, indicating potential errors in production parameters, key process steps, or important procedures. Therefore, rapid and accurate processing and identification of multi-source unstructured data in the production process is beneficial for improving product quality and production management efficiency.

[0004] In existing technologies, methods for parsing and processing unstructured data mainly fall into two categories: one is template- or rule-based matching methods, which calculate and match similarity using predefined feature templates or rule models. These methods are simple to implement and have low computational complexity, but their adaptability to changes in data form, scale differences, local defects, and noise interference is limited. In complex application scenarios, they are prone to misjudgment and missed judgment, and lack robustness. The other category is machine learning, especially deep learning, methods that achieve feature learning and classification through neural network models. These methods can achieve high recognition accuracy in complex data environments, but they usually rely on large-scale labeled data and high-computing platforms. The computational complexity of model training and inference is high, and resource consumption is large. They are difficult to deploy efficiently in resource-constrained embedded systems or real-time computing environments, and the model decision-making process lacks interpretability, limiting their application in engineering verification and industrial testing scenarios that require high reliability and traceability.

[0005] In summary, existing production data processing still has significant limitations in terms of processing efficiency, identification accuracy, and system deployability, which affects the quality of products produced and the efficiency of production management during the production process. Summary of the Invention

[0006] To address the aforementioned problems in the prior art, this application provides an unstructured data processing method, apparatus, device, and storage medium that can quickly and accurately identify whether production data is abnormal, thereby improving product quality and production management efficiency.

[0007] To achieve the above objectives, the technical solution adopted in this application is as follows: In a first aspect, this application provides a method for processing unstructured data, including: Acquire unstructured data generated during the production process; Functional basis analysis is performed on the unstructured data to obtain a feature coefficient vector characterizing the structural features of the unstructured data; The feature coefficient vector is matched with a pre-established standard feature library, and the unstructured data is determined to be abnormal based on the matching result, thus obtaining the data anomaly identification result.

[0008] The beneficial effects of this application are as follows: By introducing functional basis analysis, this application transforms unstructured data generated during the production process into feature coefficient vectors with clear mathematical meaning, thereby achieving a unified and compact expression of the inherent structural characteristics of the data. This enables rapid and accurate identification of whether production data is abnormal, thereby improving the quality of produced products and the efficiency of production management.

[0009] Optionally, the step of performing functional basis analysis on the unstructured data to obtain a feature coefficient vector characterizing the structural features of the unstructured data includes: The unstructured data is preprocessed to obtain standardized data; Construct a set of functional basis functions to characterize the structural features of the unstructured data, and express the standardized data as a weighted combination of basis functions. The set of functional basis functions contains basis functions used to characterize the elements of the data structure. The weight coefficients of the weighted combination are solved by mathematical transformation and optimization algorithm to form the feature coefficient vector.

[0010] Optionally, if the functional basis function set includes basis functions for characterizing data structure elements, then the mathematical model for expressing the standardized data as a weighted combination of basis functions is: ; In the formula, S(x,y) represents standardized data, N is the number of basis functions, b is the number of basis functions, and α... b f represents the weight coefficient of the b-th basis function. b (x,y) is the b-th basis function; Wherein, the weight coefficient α of the b-th basis function b The calculation formula is obtained by solving the problem using a regularized optimization model: ; ; In the formula, λ is the regularization coefficient, α is the weight coefficient vector, [α 1 ,α 2 ,…,α N ] T This is the transpose of the weight coefficients of the first N basis functions. Let L2 be the squared value of the function fitting error with weight coefficient α as the independent variable. It is an L1 norm.

[0011] Optionally, the step of solving the weight coefficients of the weighted combination through mathematical transformation and optimization algorithms includes: The standardized data is subjected to Hilbert transform to obtain an analytical signal, the analytical processing formula of which is: ; In the formula, For analytic symbolic signals, H[·] denotes the Hilbert transform operator, where j is the imaginary unit, satisfying j 2 =-1; The instantaneous amplitude and instantaneous phase features are extracted from the analytical signal using the following formula: ; ; In the formula, A(x,y) is the instantaneous amplitude characteristic, φ(x,y) is the instantaneous phase characteristic, and arg(·) is the argument of the complex number; By combining the instantaneous amplitude characteristics and the instantaneous phase characteristics, the weight coefficients are calculated by solving the regularized optimization model.

[0012] Optionally, the preprocessing of the unstructured data to obtain standardized data includes: The unstructured data is subjected to denoising, binarization, and line thinning to obtain standardized data; The noise reduction process employs either adaptive median filtering or Gaussian filtering. The formula for Gaussian filtering is as follows: ; In the formula, I(x,y) represents unstructured data, x is the x-coordinate of a pixel, y is the y-coordinate of a pixel, and I... d (x,y) represents the denoised data, i' represents the horizontal offset of the current pixel, j' represents the vertical offset of the current pixel, r represents the radius of the filtering window, w(i',j') represents the filtering weight, and I(x+i',y+j') represents the pixel gray value at a neighborhood offset (i',j') centered at (x,y). The binarization process employs an algorithm based on a local adaptive threshold, and the calculation formula is as follows: ; In the formula, B(x,y) represents binary data, and T(x,y) represents the local adaptive threshold. The line thinning process employs an improved Zhang-Suen thinning algorithm or a distance transformation method. The formula for the improved Zhang-Suen thinning algorithm is as follows: S(x,y)=Thinning(B(x,y)); In the formula, S(x,y) is the symbol skeleton, and Thinning(·) is the thinning operator.

[0013] Optionally, the step of matching the feature coefficient vector with a pre-established standard feature library and determining whether the unstructured data is abnormal based on the matching result to obtain a data anomaly identification result includes: Calculate the cosine similarity between the feature coefficient vector and the coefficient vectors of each template in the standard feature library. Use the category corresponding to the template with the highest similarity as the matching result for the unstructured data. The formula for calculating the cosine similarity is: ; In the formula, S(C,C) k Let C be the cosine similarity, and C be the feature coefficient vector.k Let C be the k-th template coefficient vector in the standard feature library, M be the number of template categories in the standard feature library, and C·C k Let ||·|| be the dot product of vectors, and let ||·|| be the L2 norm of the vectors. The coefficient deviation between the feature coefficient vector and the template coefficient vector matched in the standard feature library is calculated. If the coefficient deviation exceeds a preset threshold, the unstructured data is determined to be abnormal. The coefficient deviation is calculated using a normalized deviation metric formula. ; ΔC=CC m ; In the formula, E is the coefficient deviation, ΔC is the coefficient deviation vector, and C m C is the matched template coefficient vector, and C is the feature coefficient vector; If the coefficient deviation E is within the set tolerance threshold, a functional basis integrity check is performed to obtain the data anomaly identification result. The formula for the functional basis integrity check is: E fusion =βE global +(1-β)E local ; In the formula, E fusion E represents the fusion coefficient deviation. global E is due to overall shape deviation. local β represents the abnormal deviation of the local basis function, and β∈[0,1] is the weight parameter.

[0014] Optionally, the method further includes: During operation, at least one of the following optimization methods—lightweight computing architecture, memory access optimization strategy, and pipelined processing mechanism—is used for embedded algorithm optimization. The lightweight computing architecture uses fixed-point arithmetic, and the calculation formula is as follows: ; In the formula, y p For the p-th output feature, q Here, q is the fixed-point quantization function, x is the quantization precision in bits, and q is the quantization precision in bits. b For the input features, w pb The weights of the functional basis expansion coefficients; The memory access optimization strategy employs at least one of the following strategies to optimize memory access: data block fragmentation, cache priority scheduling, and memory-aligned access. The pipelined processing mechanism employs a pipelined processing architecture, which strings together the sub-steps in the feature basis modeling and matching calculation into a pipelined execution. The latency of the pipelined execution satisfies the following formula: ; In the formula, T pipeline T is the pipeline delay time. pre To preprocess runtime, T feat T represents the feature extraction runtime. basis T is the running time of basis function decomposition. coef The running time T is used to calculate the coefficients. def Let max{·} be the runtime of the problem analysis, and max{·} be the maximization function.

[0015] Secondly, this application provides an unstructured data processing apparatus, comprising: The data acquisition module is used to acquire unstructured data generated during the production process; The feature construction module is used to perform functional basis analysis on the unstructured data to obtain a feature coefficient vector characterizing the structural features of the unstructured data. The data matching module is used to perform matching calculations between the feature coefficient vector and a pre-established standard feature library, and determine whether the unstructured data is abnormal based on the matching results, thereby obtaining the data anomaly identification result.

[0016] Thirdly, this application provides a computer device, including: one or more processors, and a memory; The memory stores computer-readable instructions that, when executed by the one or more processors, perform the steps of the method as described in the first aspect.

[0017] Fourthly, this application provides a storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the method described in the first aspect.

[0018] The technical effects of the unstructured data processing apparatus provided in the second aspect, the computer device provided in the third aspect, and the storage medium provided in the fourth aspect are described in the relevant description of the unstructured data processing method provided in the first aspect. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the main flow of the unstructured data processing method according to an embodiment of this application; Figure 2 This is a schematic diagram of the structure of an unstructured data processing device according to an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a computer device according to an embodiment of this application. Detailed Implementation

[0020] To better understand the above technical solutions, exemplary embodiments of this application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application can be understood more clearly and thoroughly, and that the scope of this application can be fully conveyed to those skilled in the art.

[0021] The embodiments of this application are applied to manufacturing scenarios, such as the bioengineering equipment manufacturing industry, which generates various types of unstructured data. Existing methods based on template or rule matching, or methods based on machine learning, especially deep learning, cannot quickly and accurately identify anomalies in unstructured data, thereby affecting product quality and production management efficiency.

[0022] Therefore, in various embodiments of this application, unstructured data generated during the production process is acquired; functional basis analysis is performed on the unstructured data to obtain feature coefficient vectors characterizing the structural features of the unstructured data; the feature coefficient vectors are matched and calculated with a pre-established standard feature library, and the unstructured data is processed based on the matching results to obtain data anomaly identification results. This achieves a unified and compact expression of the inherent structural features of the data, enabling rapid and accurate identification of whether production data is abnormal, thereby improving product quality and production management efficiency.

[0023] The present application will now be described in further detail with reference to the accompanying drawings and embodiments.

[0024] This application provides an embodiment of an unstructured data processing method, such as... Figure 1 As shown, the method includes: Step S100: Obtain unstructured data generated during the production process.

[0025] Unstructured data refers to data that does not have a fixed pattern or predefined structure format. Various types of unstructured data in the entire manufacturing process include, but are not limited to, text, audio, video, images, graphic symbols, sensor time-series signals, biomedical signals, trajectory data, and other forms. In one example of this application, time-series signal data and unstructured text data are used as examples of unstructured data.

[0026] Step S200: Perform functional basis analysis on the unstructured data to obtain the feature coefficient vector that characterizes the structural features of the unstructured data.

[0027] In one embodiment, step S200 includes: Step S201: Preprocess the unstructured data to obtain standardized data.

[0028] This includes preprocessing unstructured data to obtain standardized data, including: Unstructured data is denoised, binarized, and line-thinned to obtain standardized data.

[0029] The noise reduction process employs either adaptive median filtering or Gaussian filtering. The formula for Gaussian filtering is as follows: ; In the formula, I(x,y) represents unstructured data, x is the x-coordinate of a pixel, y is the y-coordinate of a pixel, and I... d (x,y) represents the denoised data, i' represents the horizontal offset of the current pixel, j' represents the vertical offset of the current pixel, r represents the radius of the filtering window, w(i',j') represents the filtering weight, and I(x+i',y+j') represents the pixel grayscale value at a neighborhood offset (i',j') centered at (x,y).

[0030] The binarization process employs an algorithm based on local adaptive thresholding, and the calculation formula is as follows: ; In the formula, B(x,y) represents binary data, and T(x,y) represents the local adaptive threshold.

[0031] The line thinning process employs either an improved Zhang-Suen thinning algorithm or a distance transformation method. The formula for the improved Zhang-Suen thinning algorithm is as follows: S(x,y)=Thinning(B(x,y)); In the formula, S(x,y) is the symbol skeleton, and Thinning(·) is the thinning operator.

[0032] Step S202: Construct a set of functional basis functions that characterize the features of unstructured data structures. Standardized data is expressed as a weighted combination of basis functions. The set of functional basis functions contains basis functions used to characterize data structure elements.

[0033] In this context, basis functions can be any mathematical function that can characterize the basic building blocks of data. For time series signal data, basis functions can be sine waves or wavelets; for text vectors, basis functions can be word vectors; and for graphical symbols, the functional basis function set covers the basic symbolic elements commonly found in unstructured data. The graphical symbols mentioned above are just one example.

[0034] The mathematical model that expresses standardized data as a weighted combination of basis functions is as follows: ; In the formula, S(x,y) represents standardized data, N is the number of basis functions, b is the number of basis functions, and α... b f represents the weight coefficient of the b-th basis function. b (x,y) is the b-th basis function.

[0035] Wherein, the weight coefficient α of the b-th basis function b The calculation formula is obtained by solving the problem using a regularized optimization model: ; ; In the formula, λ is the regularization coefficient, α is the weight coefficient vector, [α 1 ,α 2 ,…,α N ] T This is the transpose of the weight coefficients of the first N basis functions. Let L2 be the squared value of the function fitting error with weight coefficient α as the independent variable. It is an L1 norm.

[0036] Step S203: Solve the weight coefficients of the weighted combination through mathematical transformation and optimization algorithm to form the feature coefficient vector.

[0037] The weighting coefficients of the weighted combination are solved through mathematical transformations and optimization algorithms, including: Step S2031: Perform Hilbert transform on the standardized data to obtain the analytic signal. The analytic processing formula is as follows: ; In the formula, For analytic symbolic signals, H[·] denotes the Hilbert transform operator, where j is the imaginary unit, satisfying j 2 =-1.

[0038] Step S2032: Extract instantaneous amplitude and instantaneous phase features from the analytical signal. The feature extraction formula is as follows: ; ; In the formula, A(x,y) is the instantaneous amplitude characteristic, φ(x,y) is the instantaneous phase characteristic, and arg(·) is the argument of the complex number.

[0039] Among them, the instantaneous amplitude feature is used to reflect the local intensity and contour thickness of the symbol, while the instantaneous phase feature is used to describe the geometric orientation and structural changes of the symbol.

[0040] Step S2033: Combining instantaneous amplitude characteristics and instantaneous phase characteristics, the weight coefficients are calculated by solving the regularized optimization model.

[0041] Step S300: Match the feature coefficient vector with the pre-established standard feature library, process the unstructured data according to the matching results, and obtain the data anomaly identification results.

[0042] In one embodiment, step S300 includes: Step S301: Calculate the cosine similarity between the feature coefficient vector and the coefficient vectors of each template in the standard feature library. The category corresponding to the template with the highest similarity is taken as the matching result for the unstructured data. The formula for calculating the cosine similarity is: ; In the formula, S(C,C) k Let C be the cosine similarity, and C be the feature coefficient vector. k Let C be the k-th template coefficient vector in the standard feature library, M be the number of template categories in the standard feature library, and C·C k Let be the inner product of vectors, and ||·| be the L2 norm of the vector.

[0043] Step S302: Calculate the coefficient deviation between the feature coefficient vector and the template coefficient vector matched in the standard feature library. If the coefficient deviation exceeds a preset threshold, it is determined that the unstructured data is abnormal. The coefficient deviation is calculated using a normalized deviation measurement formula. ; ΔC=CC m ; In the formula, E is the coefficient deviation, ΔC is the coefficient deviation vector, and C m C is the matched template coefficient vector, and C is the feature coefficient vector.

[0044] Step S303: If the coefficient deviation E is within the set tolerance threshold, then perform a functional basis integrity check to obtain the data anomaly identification result. The formula for the functional basis integrity check is: E fusion =βE global +(1-β)E local ; In the formula, E fusion E represents the fusion coefficient deviation. global E is due to overall shape deviation. local β represents the abnormal deviation of the local basis function, and β∈[0,1] is the weight parameter.

[0045] Integrity checks identify potential problems by determining whether there are missing or abrupt changes in key basis function coefficients.

[0046] Therefore, this embodiment maps multiple types of unstructured data in the entire equipment manufacturing process into functional basis feature coefficient vectors and performs matching and discrimination under the same analysis framework, thereby enabling rapid and accurate consistency verification, anomaly diagnosis, and intelligent processing of production data.

[0047] In one example, when processing time-series signal data, integrity checks determine whether the time-series signal data suffers from data loss, noise interference, or feature pattern distortion by analyzing whether the coefficients of corresponding key waveform components are missing or have abrupt amplitude changes. For data formats such as text semantic vectors or structured log streams, integrity checks identify potential problems such as missing content, logical contradictions, or injection of abnormal events by detecting whether the feature coefficients representing core semantics or key events deviate significantly or disappear. Through these methods, this application achieves consistency verification and problem diagnosis for multiple types of unstructured data within a unified coefficient vector analysis framework.

[0048] In one embodiment, this application further includes: During operation, at least one of the following optimization methods is used for embedded algorithm optimization: lightweight computing architecture, memory access optimization strategy, and pipelined processing mechanism.

[0049] The lightweight computing architecture uses fixed-point arithmetic, and the calculation formula is as follows: ; In the formula, y p For the p-th output feature, q Here, q is the fixed-point quantization function, x is the quantization precision in bits, and q is the quantization precision in bits. b For the input features, w pb The weights are the functional basis expansion coefficients.

[0050] Among them, the memory access optimization strategy adopts at least one of the following strategies to optimize memory access: data block fragmentation, cache priority scheduling and memory-aligned access. The pipelined processing mechanism employs a pipelined processing architecture, chaining the sub-steps in feature basis modeling and matching computation into a pipelined execution. The latency of the pipelined execution satisfies the following formula: ; In the formula, T pipeline T is the pipeline delay time. pre To preprocess runtime, T feat T represents the feature extraction runtime. basis T is the running time of basis function decomposition. coefThe running time T is used to calculate the coefficients. def Let max{·} be the runtime of the problem analysis, and max{·} be the maximization function.

[0051] In summary, the embodiments of the various methods in this application have the following technical advantages: 1. This application proposes a method for representing and processing unstructured data based on functional basis analysis. By representing data objects as a weighted combination of a small number of functional basis functions, it achieves a unified representation of heterogeneous data, improves the parsing accuracy of complex unstructured data, and ensures the accuracy of data anomaly identification.

[0052] 2. This application combines Hilbert transform with an optimized matching strategy, which can comprehensively extract the static and dynamic structural features of the data, effectively addressing issues such as scaling, deformation, local defects, or noise interference in data objects, thereby significantly enhancing the robustness and reliability of the system processing and further improving the accuracy of data anomaly identification.

[0053] 3. This application significantly reduces computational complexity and hardware resource consumption through a lightweight algorithm architecture design, combined with fixed-point arithmetic, memory access optimization and pipelined processing, enabling the method to be efficiently deployed on embedded platforms and meet the requirements of low-power real-time processing.

[0054] 4. This application has good versatility and scalability, and can be widely used in fields such as electronic design automation, industrial visual inspection, and intelligent hardware monitoring, providing reliable technical support for automated quality inspection and data processing, and has broad prospects for industrial application.

[0055] Therefore, this application can achieve rapid and accurate consistency verification, anomaly diagnosis and intelligent processing of production data, thereby improving the digitalization level of equipment manufacturing process, product quality, production management efficiency and application scope.

[0056] In one embodiment, such as Figure 2 As shown, this application also provides an unstructured data processing apparatus, including: Data acquisition module 201 is used to acquire unstructured data generated during the production process; The feature construction module 202 is used to perform functional basis analysis on unstructured data to obtain feature coefficient vectors that characterize the structural features of unstructured data. The data matching module 203 is used to perform matching calculations between the feature coefficient vector and the pre-established standard feature library, and to determine whether the unstructured data is abnormal based on the matching results, thereby obtaining the data anomaly identification results.

[0057] In one embodiment, such as Figure 3 As shown, this application also provides a computer device 300, comprising: The communication interface 301 allows for information exchange with other devices or network nodes.

[0058] One or more processors 302 are connected to a communication interface 301 to enable information interaction with other devices or network nodes, and to execute the methods provided by one or more technical solutions in the above embodiments when running computer programs.

[0059] Memory 303 is used to store computer-readable instructions that can be executed on processor 302. When executed by one or more processors 302, the computer-readable instructions perform the steps of the software development method as described in the above embodiments.

[0060] The computer device 300 in this embodiment only shows a block diagram of a portion of the structure related to the present application solution, and does not constitute a limitation on the computer device to which the present application solution is applied. A specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements, such as power supply, input / output interfaces, etc. Furthermore, the computer device 300 of this embodiment can operate on an operating system stored in memory 303, such as Windows Server™, Mac OS X™, Unix™, Linux™, Free BSD™, or similar.

[0061] In one embodiment, this application also provides a storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the software development method as described in the above embodiments.

[0062] The computer-readable instructions in the aforementioned computer device 300 and storage medium may be application programs.

[0063] Furthermore, the specific descriptions of the technical effects and steps corresponding to the unstructured data processing apparatus, a computer device 300, and a storage medium in the above embodiments are all based on the relevant descriptions in the embodiments of the unstructured data processing method.

[0064] Since the systems / devices described in the above embodiments of this application are systems / devices used to implement the methods of the above embodiments of this application, those skilled in the art can understand the specific structure and modifications of the system / devices based on the methods described in the above embodiments of this application, and therefore will not be repeated here. All systems / devices used in the methods of the above embodiments of this application fall within the scope of protection of this application.

[0065] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0066] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions.

[0067] It should be noted that any reference numerals placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The words "a" or "an" preceding a component do not exclude the presence of a plurality of such components. This application can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In claims that enumerate several means, several of these means may be embodied by the same hardware. The use of the terms first, second, third, etc., is merely for convenience of expression and does not indicate any order. These terms can be understood as part of the component names.

[0068] Furthermore, it should be noted that in the description of this specification, the terms "one embodiment," "some embodiments," "embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0069] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the claims should be interpreted to include the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0070] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if these modifications and variations fall within the scope of the claims of this application and their equivalents, then this application should also include these modifications and variations.

Claims

1. An unstructured data processing method, characterized in that, include: Acquire unstructured data generated during the production process; Functional basis analysis is performed on the unstructured data to obtain a feature coefficient vector characterizing the structural features of the unstructured data; The feature coefficient vector is matched with a pre-established standard feature library, and the unstructured data is determined to be abnormal based on the matching result, thus obtaining the data anomaly identification result.

2. The method according to claim 1, characterized in that, The step of performing functional basis analysis on the unstructured data to obtain a feature coefficient vector characterizing the structural features of the unstructured data includes: The unstructured data is preprocessed to obtain standardized data; Construct a set of functional basis functions to characterize the structural features of the unstructured data, and express the standardized data as a weighted combination of basis functions. The set of functional basis functions contains basis functions used to characterize the elements of the data structure. The weight coefficients of the weighted combination are solved by mathematical transformation and optimization algorithm to form the feature coefficient vector.

3. The method according to claim 2, characterized in that, The mathematical model that expresses the standardized data as a weighted combination of basis functions is as follows: ； In the formula, S(x,y) represents standardized data, N is the number of basis functions, b is the number of basis functions, and α... b f represents the weight coefficient of the b-th basis function. b (x,y) is the b-th basis function; Wherein, the weight coefficient α of the b-th basis function b The calculation formula is obtained by solving the problem using a regularized optimization model: ； ； In the formula, λ is the regularization coefficient, α is the weight coefficient vector, [α 1 ,α 2 ,…,α N ] T This is the transpose of the weight coefficients of the first N basis functions. Let L2 be the squared value of the function fitting error with weight coefficient α as the independent variable. It is an L1 norm.

4. The method according to claim 2, characterized in that, The process of solving the weight coefficients of the weighted combination through mathematical transformations and optimization algorithms includes: The standardized data is subjected to Hilbert transform to obtain an analytical signal, and the analytical processing formula is as follows: ； In the formula, For analytic symbolic signals, H[·] denotes the Hilbert transform operator, where j is the imaginary unit, satisfying j 2 =-1; The instantaneous amplitude and instantaneous phase features are extracted from the analytical signal using the following formula: ； ； In the formula, A(x,y) is the instantaneous amplitude characteristic, φ(x,y) is the instantaneous phase characteristic, and arg(·) is the argument of the complex number; By combining the instantaneous amplitude characteristics and the instantaneous phase characteristics, the weight coefficients are calculated by solving the regularized optimization model.

5. The method according to claim 3, characterized in that, The preprocessing of the unstructured data to obtain standardized data includes: The unstructured data is subjected to denoising, binarization, and line thinning to obtain standardized data. The noise reduction process employs either adaptive median filtering or Gaussian filtering. The formula for Gaussian filtering is as follows: ； In the formula, I(x,y) represents unstructured data, x is the x-coordinate of a pixel, y is the y-coordinate of a pixel, and I... d (x,y) represents the denoised data, i' represents the horizontal offset of the current pixel, j' represents the vertical offset of the current pixel, r represents the radius of the filtering window, w(i',j') represents the filtering weight, and I(x+i',y+j') represents the pixel gray value at a neighborhood offset (i',j') centered at (x,y). The binarization process employs an algorithm based on a local adaptive threshold, and the calculation formula is as follows: ； In the formula, B(x,y) represents binary data, and T(x,y) represents the local adaptive threshold. The line thinning process employs an improved Zhang-Suen thinning algorithm or a distance transformation method. The formula for the improved Zhang-Suen thinning algorithm is as follows: S(x,y)=Thinning(B(x,y)); In the formula, S(x,y) is the symbol skeleton, and Thinning(·) is the thinning operator.

6. The method according to any one of claims 1 to 5, characterized in that, The step of matching the feature coefficient vector with a pre-established standard feature library and determining whether the unstructured data is abnormal based on the matching result to obtain a data anomaly identification result includes: Calculate the cosine similarity between the feature coefficient vector and the coefficient vectors of each template in the standard feature library. Use the category corresponding to the template with the highest similarity as the matching result for the unstructured data. The formula for calculating the cosine similarity is: ； In the formula, S(C,C) k Let C be the cosine similarity, and C be the feature coefficient vector. k Let C be the k-th template coefficient vector in the standard feature library, M be the number of template categories in the standard feature library, and C·C k Let ||·|| be the dot product of vectors, and let ||·|| be the L2 norm of the vectors. The coefficient deviation between the feature coefficient vector and the template coefficient vector matched in the standard feature library is calculated. If the coefficient deviation exceeds a preset threshold, the unstructured data is determined to be abnormal. The coefficient deviation is calculated using a normalized deviation metric formula. ； ΔC=C-C m ; In the formula, E is the coefficient deviation, ΔC is the coefficient deviation vector, and C m C is the matched template coefficient vector, and C is the feature coefficient vector; If the coefficient deviation E is within the set tolerance threshold, a functional basis integrity check is performed to obtain the data anomaly identification result. The formula for the functional basis integrity check is: E fusion =βE global +(1-β)E local ; In the formula, E fusion E represents the fusion coefficient deviation. global E is due to overall shape deviation. local β represents the abnormal deviation of the local basis function, and β∈[0,1] is the weight parameter.

7. The method according to any one of claims 1 to 5, characterized in that, The method further includes: During operation, at least one of the following optimization methods—lightweight computing architecture, memory access optimization strategy, and pipelined processing mechanism—is used for embedded algorithm optimization. The lightweight computing architecture uses fixed-point arithmetic, and the calculation formula is as follows: ； In the formula, y p For the p-th output feature, q Here, q is the fixed-point quantization function, x is the quantization precision in bits, and q is the quantization precision in bits. b For the input features, w pb The weights of the functional basis expansion coefficients; The memory access optimization strategy employs at least one of the following strategies to optimize memory access: data block fragmentation, cache priority scheduling, and memory-aligned access. The pipelined processing mechanism employs a pipelined processing architecture, which strings together the sub-steps in the feature basis modeling and matching calculation into a pipelined execution. The latency of the pipelined execution satisfies the following formula: ； In the formula, T pipeline T is the pipeline delay time. pre To preprocess runtime, T feat T is the feature extraction runtime. basis Let T be the running time of the basis function decomposition. coef The running time T is used to calculate the coefficients. def Let max{·} be the runtime of the problem analysis, and max{·} be the maximization function.

8. An unstructured data processing device, characterized in that, include: The data acquisition module is used to acquire unstructured data generated during the production process; The feature construction module is used to perform functional basis analysis on the unstructured data to obtain a feature coefficient vector characterizing the structural features of the unstructured data. The data matching module is used to perform matching calculations between the feature coefficient vector and a pre-established standard feature library, and determine whether the unstructured data is abnormal based on the matching results, thereby obtaining the data anomaly identification result.

9. A computer device, characterized in that, include: One or more processors, and memory; The memory stores computer-readable instructions that, when executed by the one or more processors, perform the steps of the method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the method as described in any one of claims 1 to 7.

Images