Method and apparatus for processing unstructured data, and device and medium

Abstract

The present application relates to the technical field of data analysis. Provided are a method and apparatus for processing unstructured data, and a device and a medium. The method comprises: acquiring unstructured data related to a financial risk; preprocessing the unstructured data to obtain preprocessed unstructured data; performing feature extraction on the preprocessed unstructured data to obtain target features; performing uncertainty processing and fuzziness processing on the target features to obtain an uncertainty analysis result and a fuzziness processing result; performing contextual analysis on the target features to obtain contextual association information; and on the basis of the uncertainty analysis result, the fuzziness processing result and the contextual association information, evaluating the financial risk to obtain a financial risk evaluation result. In the present application, the actual condition of a financial risk can be more accurately reflected by means of uncertainty processing and fuzziness processing, and potential associations in data can be identified by means of contextual analysis, thereby improving the accuracy of unstructured data processing.

Description

Unstructured data processing method, device, equipment and medium Technical Field

[0001] The present application relates to the field of data analysis technology, and more specifically, to a method, apparatus, device, and medium for processing unstructured data. Background Art

[0002] With the rapid development of financial markets, financial risk control has become increasingly important. Traditional financial risk control methods primarily rely on structured data. However, unstructured data, which contains a wealth of information such as market sentiment and public opinion trends, is equally valuable for financial risk control. Structured data refers to data stored in a predetermined format with a clear logical structure, typically in tabular form, consisting of rows (records) and columns (fields). Unstructured data refers to data without a predefined data model or structure, typically containing large amounts of information in the form of text, images, audio, video, and other formats.

[0003] However, when processing unstructured data, existing technologies have difficulty obtaining accurate fuzzy logic, implicit information, and contextual information due to the complex and changeable limitations of unstructured data related to financial risks, which in turn reduces the accuracy of unstructured data processing.

[0004] Therefore, how to improve the accuracy of unstructured data processing is an urgent problem that needs to be solved. Summary of the Invention

[0005] In view of the above problems existing in the prior art, the purpose of this application is to propose an unstructured data processing method, device, equipment and medium to at least solve the technical problem of how to improve the accuracy of unstructured data processing.

[0006] To achieve the above-mentioned and other related purposes, the present application provides a method for processing unstructured data, the method comprising:

[0007] Acquire unstructured data related to financial risks based on multiple data acquisition channels;

[0008] Preprocessing the unstructured data to obtain preprocessed unstructured data;

[0009] Performing feature extraction on the preprocessed unstructured data to obtain target features;

[0010] Performing uncertainty processing and fuzziness processing on the target feature to obtain uncertainty analysis results and fuzziness processing results;

[0011] Performing context analysis on the target feature to obtain context-related information;

[0012] Based on the uncertainty analysis result, the fuzziness processing result and the context association information, the financial risk is evaluated to obtain a financial risk evaluation result.

[0013] In one embodiment of the present application, the multiple data acquisition channels include social media, new websites, online forums, customer feedback channels, and third-party data suppliers.

[0014] In one embodiment of the present application, preprocessing the unstructured data to obtain preprocessed unstructured data includes:

[0015] removing noise from the unstructured data to obtain first preprocessed data, detecting missing information in the first preprocessed data, and performing interpolation processing or image restoration processing on the missing information to obtain second preprocessed data;

[0016] detecting text data in the second preprocessed data to obtain words or sentences, and assigning attention weights to the words or sentences to obtain preprocessed text data;

[0017] Allocating attention weights on the channel dimension to the image data in the second preprocessed data to obtain preprocessed image data;

[0018] For the audio data in the second preprocessed data, attention weights are allocated in the frequency domain to obtain preprocessed audio data.

[0019] In one embodiment of the present application, extracting features from the preprocessed unstructured data to obtain target features includes:

[0020] Segmenting text data in the preprocessed unstructured data into words or phrases, removing words or phrases that do not carry information, and converting remaining words into lexical features;

[0021] Scaling and normalizing the image data of the preprocessed unstructured data, and extracting image features from the image data through a convolutional neural network;

[0022] Performing noise reduction and equalization processing on the audio data in the preprocessed unstructured data, and extracting audio features in the audio data through Mel-frequency cepstral coefficients;

[0023] Among them, the vocabulary features, image features, and audio features are included in the target features.

[0024] In one embodiment of the present application, uncertainty processing and fuzziness processing are performed on the target feature to obtain uncertainty analysis results and fuzziness processing results, including:

[0025] Constructing an uncertainty analysis model through a Bayesian network and a membership function, transmitting the target feature to the uncertainty analysis model, and quantifying the output result of the uncertainty analysis model into a confidence interval and credibility to obtain the uncertainty analysis result;

[0026] Identify the fuzzy expression in the target feature to obtain a fuzzy set, analyze the fuzzy set through fuzzy logic operation to obtain a fuzzy logic operation result, and perform fuzzy reasoning on the fuzzy set based on the fuzzy logic operation result to obtain the fuzziness processing result.

[0027] In one embodiment of the present application, performing context analysis on the target feature to obtain contextual information includes:

[0028] Transmitting the lexical features to a pre-trained sentiment analysis model to obtain sentiment analysis results, wherein the sentiment analysis model is constructed based on a convolutional neural network, and the sentiment analysis results include positive sentiment tendencies, negative sentiment tendencies, and neutral sentiment tendencies;

[0029] Transmitting the vocabulary features to a pre-trained topic extraction model to obtain key topics and concepts;

[0030] Performing co-occurrence analysis and dependency analysis on the lexical features to obtain lexical context information corresponding to the key topics and concepts;

[0031] Transmitting the image features to a pre-trained image recognition model, determining a plurality of objects corresponding to the image features, location information of each of the objects, and scene information, and obtaining image context information based on the plurality of objects, location information, and scene information;

[0032] The vocabulary context information and the image context information are transmitted to a pre-built joint embedding model to obtain the context association information.

[0033] In one embodiment of the present application, the financial risk is assessed based on the uncertainty analysis result, the fuzziness processing result, and the contextual association information to obtain a financial risk assessment result, including:

[0034] Acquiring structured data related to the financial risk through the multiple data acquisition channels;

[0035] fusing the structured data with the uncertainty analysis result, the fuzziness processing result, and the context association information to obtain fused data;

[0036] The fused data is transmitted to a pre-trained financial risk assessment model to perform financial risk trend assessment to obtain the financial risk assessment result.

[0037] In one embodiment of the present application, an unstructured data processing device is further provided, the device comprising:

[0038] A data acquisition module, used to acquire unstructured data related to financial risks based on multiple data acquisition channels;

[0039] A preprocessing module, configured to preprocess the unstructured data to obtain preprocessed unstructured data;

[0040] A feature extraction module is used to extract features from the preprocessed unstructured data to obtain target features;

[0041] An uncertainty processing and fuzziness processing module, used for performing uncertainty processing and fuzziness processing on the target feature to obtain uncertainty analysis results and fuzziness processing results;

[0042] A context analysis module, configured to perform context analysis on the target feature to obtain contextual association information;

[0043] The financial risk assessment module is used to assess the financial risk based on the uncertainty analysis result, the fuzziness processing result and the context association information to obtain a financial risk assessment result.

[0044] In one embodiment of the present application, a computer-readable storage medium is further provided, wherein the computer-readable storage medium includes a stored computer program, wherein the computer program executes the above-mentioned unstructured data processing method when running.

[0045] In one embodiment of the present application, an electronic device is further provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-mentioned unstructured data processing method through the computer program.

[0046] Beneficial effects of the present invention:

[0047] First, based on multiple data acquisition channels, unstructured data related to financial risks are acquired; then, the unstructured data are preprocessed to obtain preprocessed unstructured data; then, feature extraction is performed on the preprocessed unstructured data to obtain target features; then, uncertainty processing and fuzziness processing are performed on the target features to obtain uncertainty analysis results and fuzziness processing results; then, context analysis is performed on the target features to obtain context-related information; finally, based on the uncertainty analysis results, the fuzziness processing results and the context-related information, the financial risks are evaluated to obtain financial risk evaluation results. In this application, collecting data through multiple data acquisition channels can cover a wider range of information sources and increase the diversity and comprehensiveness of the data; preprocessing unstructured data can eliminate noise, missing values ​​and redundant information in the data and improve data quality; by performing feature extraction on the preprocessed unstructured data, the unstructured raw data can be converted into structured feature vectors, reducing data dimensions and computational complexity. Feature extraction can also highlight key information and improve the data's ability to represent financial risks; uncertainty processing can quantify the likelihood and impact of risk occurrence, while fuzzy processing can handle ambiguous expressions and unclear boundaries in the data. These two processing methods can more accurately reflect the actual situation of financial risks and improve the accuracy of unstructured data processing; through context analysis, potential correlations in the data can be identified, avoiding the limitations of a single perspective and further improving the accuracy of unstructured data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0049] FIG1 is a schematic diagram of an application environment of an unstructured data processing method shown in an exemplary embodiment of the present application;

[0050] FIG2 is a flow chart of a method for processing unstructured data according to an exemplary embodiment of the present application;

[0051] FIG3 is a schematic diagram of an unstructured data processing apparatus shown in an exemplary embodiment of the present application;

[0052] FIG4 is a schematic structural diagram of an electronic device according to an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0054] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0055] In one embodiment of the present application, a method for processing unstructured data is provided. Optionally, as an optional implementation, the unstructured data processing method may be applied in an environment as shown in FIG1 . FIG1 is a schematic diagram illustrating an application environment of the unstructured data processing method according to an exemplary embodiment of the present application. Referring to FIG1 , a data processing terminal 101 may be, but is not limited to, a host computer, desktop computer, laptop computer, etc. with local computing capabilities. The data processing terminal 101 may communicate with a data acquisition terminal 102 and a server 103 via a network, and the server 103 may perform operations on a database, such as writing or reading data. The data processing terminal 101 may include, but is not limited to, a human-computer interaction screen, a processor, and a memory. The human-computer interaction screen may be, but is not limited to, used to display the results of the unstructured data processing. The processor may be, but is not limited to, used to respond to the human-computer interaction operations, execute corresponding operations, or generate corresponding instructions and send the generated instructions to the server 103. The memory is used to store relevant data, such as unstructured data related to financial risks.

[0056] As an optional method, data can be collected through the data collection terminal 102, for example, unstructured data related to financial risks can be collected and preprocessed.

[0057] As an optional manner, the following steps in the unstructured data processing method may be performed on the data processing terminal 101:

[0058] Acquire unstructured data related to financial risks based on multiple data acquisition channels;

[0059] Preprocessing the unstructured data to obtain preprocessed unstructured data;

[0060] Performing feature extraction on the preprocessed unstructured data to obtain target features;

[0061] Performing uncertainty processing and fuzziness processing on the target feature to obtain uncertainty analysis results and fuzziness processing results;

[0062] Performing context analysis on the target feature to obtain context-related information;

[0063] Based on the uncertainty analysis result, the fuzziness processing result and the context association information, the financial risk is evaluated to obtain a financial risk evaluation result.

[0064] As an optional method, the server 103 may obtain the financial risk assessment result transmitted by the data processing terminal 101 and then perform subsequent processing.

[0065] The above method collects data through multiple data acquisition channels, which can cover a wider range of information sources and increase the diversity and comprehensiveness of data; preprocessing unstructured data can eliminate noise, missing values ​​and redundant information in the data and improve data quality; by extracting features from the preprocessed unstructured data, the unstructured raw data can be converted into structured feature vectors, reducing data dimensions and computational complexity. Feature extraction can also highlight key information and improve the data's ability to represent financial risks; uncertainty processing can quantify the possibility and impact of risks, while fuzzy processing can deal with ambiguous statements and unclear boundaries in the data. These two processing methods can more accurately reflect the actual situation of financial risks and improve the accuracy of unstructured data processing; through context analysis, potential correlations in the data can be identified, avoiding the limitations of a single perspective and further improving the accuracy of unstructured data processing.

[0066] Optionally, in this embodiment, the data processing terminal 101 may be a terminal device configured with a target client, which may include but is not limited to at least one of the following: a laptop computer, a tablet computer, a handheld computer, a PAD, a desktop computer, etc. The target client may be a video client, an instant messaging client, a browser client, an education client, or other client that supports providing unstructured data processing applications. The network may include but is not limited to: a wired network, a wireless network, wherein the wired network includes: a local area network, a metropolitan area network, and a wide area network, and the wireless network includes: Bluetooth, WIFI, and other networks that implement wireless communication. The server may be a single server, or a server cluster consisting of multiple servers, or a cloud server. The above is only an example, and this embodiment does not impose any limitation on this.

[0067] As an optional example, this embodiment does not limit the execution subject of the above-mentioned unstructured data processing method. Some or all steps of the above-mentioned unstructured data processing method can be executed on the data processing terminal 101. For example, when the data processing terminal 101 is a desktop computer, some or all steps of the above-mentioned unstructured data processing method can be executed on the desktop computer.

[0068] In one embodiment of the present application, a method for processing unstructured data is provided. FIG2 is a flow chart of the method for processing unstructured data according to an exemplary embodiment of the present application. Referring to FIG2 , the method for processing unstructured data includes the following steps S210 to S260:

[0069] In step S210 , unstructured data related to financial risks is acquired based on multiple data acquisition channels.

[0070] Among them, unstructured data related to financial risks are collected from multiple data acquisition channels, including social media, news reports, market reports, information released by the government and regulatory agencies, internal reports and communication records.

[0071] In step S220, the unstructured data is preprocessed to obtain preprocessed unstructured data.

[0072] Preprocessing unstructured data is a key step in data analysis, aiming to improve data quality and make it more suitable for subsequent feature extraction and modeling. Preprocessing text data in unstructured data can include data cleaning and standardization to obtain clean, segmented, and standardized text, as well as word vector representations or context-sensitive word vector representations. Preprocessing image data in unstructured data can include image standardization and data augmentation. It can also segment images into different regions to extract specific areas of interest, remove background information from images while retaining the main body, and use edge detection algorithms to enhance edge information in images. Ultimately, images of uniform size and color standardization are obtained, as well as data-enhanced image collections. Preprocessing audio data in unstructured data can include audio data standardization and data augmentation. It can also use filters to remove background noise from the audio, and segment the audio signal into short time windows to facilitate subsequent feature extraction. Ultimately, audio of uniform sampling rate and volume standardization is obtained, as well as data-enhanced audio collections.

[0073] In step S230, feature extraction is performed on the preprocessed unstructured data to obtain target features.

[0074] Feature extraction from preprocessed unstructured data is a key step in data analysis and modeling. The goal of feature extraction is to extract representations from raw data that effectively reflect the essential characteristics of the data. Target features can include text features, image features, and audio features. Text data features can include word vectors, context-sensitive word vectors, sentence vectors, topic distributions, and sentiment scores; image features can include manual features, deep learning features, object detection results, and image segmentation results; and audio features can include time domain features, frequency domain features, time-frequency features, and deep learning features.

[0075] In step S240, uncertainty processing and fuzziness processing are performed on the target feature to obtain uncertainty analysis results and fuzziness processing results.

[0076] Uncertainty and fuzziness processing of target features is designed to better manage and express uncertainty and fuzziness in unstructured data. The results of uncertainty analysis and fuzziness processing can be used to more accurately assess financial risk, helping decision makers better understand and address uncertainty. Furthermore, these results can be used to build more robust forecasting models, improving the effectiveness of financial risk management.

[0077] In step S250, context analysis is performed on the target feature to obtain context association information.

[0078] Contextual analysis of target features is a crucial step in understanding relationships between data and uncovering potential patterns and insights. Contextual information refers to the associations and dependencies between features and external context (such as time, location, and user behavior). This type of analysis is particularly important for improving the accuracy of financial risk assessments, as it can reveal the complex factors and dynamic changes behind financial risks.

[0079] In step S260, the financial risk is assessed based on the uncertainty analysis result, the fuzziness processing result and the context association information to obtain a financial risk assessment result.

[0080] For example, for credit risk assessment, we can first assess the uncertainty of the borrower's repayment ability, income level and other characteristics, then process the ambiguous information in the application form, such as income range, years of work experience, etc., and then analyze the borrower's financial report, credit history, social media comments, etc., extract keywords, identify key entities and relationships, and finally use the above information to train the credit risk assessment model to generate the borrower's risk score.

[0081] For example, for market risk monitoring, you can first monitor news reports, social media posts, etc. to assess the uncertainty of market fluctuations, then process the vague information in the news reports, such as market expectations, expert forecasts, etc., and then analyze the entity relationships and emotional tendencies in the news reports to assess market sentiment. Finally, use the above information to train the market risk monitoring model to generate a market risk score.

[0082] For example, for operational risk management, we can first assess the uncertainty of operational risk events, such as system failures and human errors, and then process the ambiguous information in internal reports, such as event descriptions and impact levels. We can then analyze the key entities and event relationships in the internal reports to assess operational risks. Finally, we can use the above information to train the operational risk management model and generate a risk score for operational risks.

[0083] By adopting the above-mentioned embodiments provided by the present application, data is collected through multiple data acquisition channels, which can cover a wider range of information sources and increase the diversity and comprehensiveness of the data; preprocessing unstructured data can eliminate noise, missing values ​​and redundant information in the data and improve data quality; by performing feature extraction on the preprocessed unstructured data, the unstructured raw data can be converted into structured feature vectors, reducing data dimensions and computational complexity. Feature extraction can also highlight key information and improve the data's ability to characterize financial risks; uncertainty processing can quantify the possibility and impact of risks, while fuzzy processing can handle ambiguous expressions and unclear boundaries in the data. These two processing methods can more accurately reflect the actual situation of financial risks and improve the accuracy of unstructured data processing; through context analysis, potential correlations in the data can be identified, avoiding the limitations of a single perspective and further improving the accuracy of unstructured data processing.

[0084] In one embodiment of the present application, the multiple data acquisition channels include social media, new websites, online forums, customer feedback channels, and third-party data suppliers.

[0085] In one embodiment of the present application, preprocessing the unstructured data to obtain preprocessed unstructured data includes:

[0086] removing noise from the unstructured data to obtain first preprocessed data, detecting missing information in the first preprocessed data, and performing interpolation processing or image restoration processing on the missing information to obtain second preprocessed data;

[0087] detecting text data in the second preprocessed data to obtain words or sentences, and assigning attention weights to the words or sentences to obtain preprocessed text data;

[0088] Allocating attention weights on the channel dimension to the image data in the second preprocessed data to obtain preprocessed image data;

[0089] For the audio data in the second preprocessed data, attention weights are allocated in the frequency domain to obtain preprocessed audio data.

[0090] In this embodiment, by removing the noise in the unstructured data, the purity of the data can be significantly improved, the interference factors in subsequent processing can be reduced, and it helps to improve the accuracy of feature extraction and modeling. Interpolating or image repairing the missing information can fill the gaps in the data, make the data more complete, and thus improve the effect of model training. By assigning attention weights to words, sentences, image channels, and audio frequency domains, the important information in the data can be enhanced, unnecessary details can be reduced, and it helps to improve the model's ability to capture key information. The attention mechanism can make the model more focused on the important parts of the input data, thereby improving the learning efficiency and generalization ability of the model.

[0091] In one embodiment of the present application, the feature extraction of the preprocessed unstructured data to obtain target features includes:

[0092] Segmenting the text data in the preprocessed unstructured data into words or phrases, removing the words or phrases that do not carry information, and converting the remaining vocabulary into vocabulary features;

[0093] Scaling and normalizing the image data in the preprocessed unstructured data, and extracting image features in the image data through a convolutional neural network;

[0094] Performing noise reduction and equalization on the audio data in the preprocessed unstructured data, and extracting audio features in the audio data through Mel-frequency cepstral coefficients;

[0095] Among them, the vocabulary features, image features, and audio features are included in the target features.

[0096] In this embodiment, segmenting the text data in the preprocessed unstructured data into words or phrases can break down the text data into manageable units, laying a foundation for subsequent analysis. Removing the words or phrases that do not carry information, such as removing stop words (such as "of", "and", etc.), can reduce the noise of the data and improve the efficiency and effect of subsequent processing. Converting the remaining vocabulary into features can quantify the text data and make it available for machine learning models. Scaling and normalizing the image data in the preprocessed unstructured data ensures that the image data has a unified size and ratio, facilitating model processing, and at the same time, normalization can accelerate the convergence speed of model training. By removing background noise and adjusting the volume, the quality of the audio data can be improved, making the audio features more prominent and clear.

[0097] In one embodiment of the present application, performing uncertainty processing and ambiguity processing on the target features to obtain uncertainty analysis results and ambiguity processing results includes:

[0098] Constructing an uncertainty analysis model through a Bayesian network and a membership function, transmitting the target feature to the uncertainty analysis model, and quantifying the output result of the uncertainty analysis model into a confidence interval and credibility to obtain the uncertainty analysis result;

[0099] Identify the fuzzy expression in the target feature to obtain a fuzzy set, analyze the fuzzy set through fuzzy logic operation to obtain a fuzzy logic operation result, and perform fuzzy reasoning on the fuzzy set based on the fuzzy logic operation result to obtain the fuzziness processing result.

[0100] In this embodiment, the uncertainty analysis model constructed using a Bayesian network can quantify the uncertainty of target features and present it in the form of confidence intervals and credibility. This helps decision makers more accurately assess risks, predict future trends, and make more robust decisions. Bayesian networks can handle multi-source information fusion and uncertainty propagation. Even when input data is noisy or inaccurate, they can still produce relatively reliable results through probabilistic reasoning. This capability makes the model more robust in the complex and volatile financial environment. The quantitative representation of confidence intervals and credibility allows risk managers to more clearly understand the scope and likelihood of risk, thereby formulating more refined risk management strategies.

[0101] In this embodiment, financial data often contains fuzzy expressions (such as "high," "low," and "medium"), which are difficult to represent using traditional precise numerical values. Fuzzy logic can process this fuzzy information by identifying fuzzy expressions and constructing fuzzy sets, thereby preserving the integrity and richness of the data. Fuzzy logic operations and fuzzy reasoning can simulate the human fuzzy thinking process, analyzing and reasoning about fuzzy sets. This capability enables the system to handle more complex and uncertain financial problems, such as assessing market trends and predicting customer demand. Fuzzy processing allows the system to make flexible decisions within a certain range, rather than relying solely on precise numerical results. This flexibility enables the system to better adapt to the rapid changes and uncertainties of the financial market.

[0102] In one embodiment of the present application, performing context analysis on the target feature to obtain contextual association information includes:

[0103] Transmitting the lexical features to a pre-trained sentiment analysis model to obtain sentiment analysis results, wherein the sentiment analysis model is constructed based on a convolutional neural network, and the sentiment analysis results include positive sentiment tendencies, negative sentiment tendencies, and neutral sentiment tendencies;

[0104] Transmitting the vocabulary features to a pre-trained topic extraction model to obtain key topics and concepts;

[0105] Performing co-occurrence analysis and dependency analysis on the lexical features to obtain lexical context information corresponding to the key topics and concepts;

[0106] Transmitting the image features to a pre-trained image recognition model, determining a plurality of objects corresponding to the image features, location information of each of the objects, and scene information, and obtaining image context information based on the plurality of objects, location information, and scene information;

[0107] The vocabulary context information and the image context information are transmitted to a pre-built joint embedding model to obtain the context association information.

[0108] In this embodiment, a sentiment analysis model (based on a convolutional neural network) processes lexical features to identify positive, negative, and neutral sentiment within a text. This helps understand the emotional tone of the text and provides a basis for subsequent sentiment analysis and sentiment assessment. A pre-trained topic extraction model processes lexical features to extract key themes and concepts within the text. This helps understand the core content of the text and supports subsequent topic analysis and information extraction. Co-occurrence analysis and dependency analysis capture the associations between words, which is crucial for understanding context within the text. Co-occurrence analysis reveals which words frequently appear together, while dependency analysis reveals the grammatical relationships between words. Co-occurrence and dependency analysis enhance the model's ability to understand text, particularly when processing complex financial text, enabling a better understanding of the logical and semantic relationships between words. A pre-trained image recognition model can identify multiple objects and their location information within an image, which is crucial for understanding the scene and content within the image. Image recognition models not only recognize objects but also extract contextual information. For example, in financial risk assessment, they can identify financial-related scenes (such as banks and exchanges). By fusing lexical context information and image context information through a joint embedding model, we can achieve the complementarity of text and image information, thereby obtaining more comprehensive contextual association information.

[0109] In one embodiment of the present application, the financial risk is assessed based on the uncertainty analysis result, the fuzziness processing result, and the contextual association information to obtain a financial risk assessment result, including:

[0110] Acquiring structured data related to the financial risk through the multiple data acquisition channels;

[0111] fusing the structured data with the uncertainty analysis result, the fuzziness processing result, and the context association information to obtain fused data;

[0112] The fused data is transmitted to a pre-trained financial risk assessment model to perform financial risk trend assessment to obtain the financial risk assessment result.

[0113] It can be seen from the above embodiments that collecting data through multiple data acquisition channels can cover a wider range of information sources and increase the diversity and comprehensiveness of data; preprocessing unstructured data can eliminate noise, missing values ​​and redundant information in the data and improve data quality; by extracting features from the preprocessed unstructured data, the unstructured raw data can be converted into structured feature vectors, reducing data dimensions and computational complexity. Feature extraction can also highlight key information and improve the data's ability to characterize financial risks; uncertainty processing can quantify the likelihood and impact of risks, while fuzzy processing can handle ambiguous expressions and unclear boundaries in the data. These two processing methods can more accurately reflect the actual situation of financial risks and improve the accuracy of unstructured data processing; through context analysis, potential correlations in the data can be identified, avoiding the limitations of a single perspective and further improving the accuracy of unstructured data processing.

[0114] In one embodiment of the present application, an unstructured data processing device is further provided. FIG3 is a schematic diagram of an unstructured data processing device shown in an exemplary embodiment of the present application. Referring to FIG3 , the device includes:

[0115] A data acquisition module 301 is used to acquire unstructured data related to financial risks based on multiple data acquisition channels;

[0116] A preprocessing module 302 is configured to preprocess the unstructured data to obtain preprocessed unstructured data;

[0117] A feature extraction module 303 is used to extract features from the pre-processed unstructured data to obtain target features;

[0118] The uncertainty processing and fuzziness processing module 304 is used to perform uncertainty processing and fuzziness processing on the target feature to obtain uncertainty analysis results and fuzziness processing results;

[0119] A context analysis module 305 is used to perform context analysis on the target feature to obtain contextual association information;

[0120] The financial risk assessment module 306 is configured to assess the financial risk based on the uncertainty analysis result, the fuzziness processing result and the context association information to obtain a financial risk assessment result.

[0121] The unstructured data processing device in the embodiment of the present application collects data through multiple data acquisition channels, can cover a wider range of information sources, and increase the diversity and comprehensiveness of the data; preprocessing the unstructured data can eliminate noise, missing values ​​and redundant information in the data, and improve the data quality; by extracting features from the preprocessed unstructured data, the unstructured raw data can be converted into structured feature vectors, reducing the data dimension and computational complexity. Feature extraction can also highlight key information and improve the data's ability to characterize financial risks; uncertainty processing can quantify the possibility and impact of risks, while fuzzy processing can handle ambiguous expressions and unclear boundaries in the data. These two processing methods can more accurately reflect the actual situation of financial risks and improve the accuracy of unstructured data processing; through context analysis, potential correlations in the data can be identified, avoiding the limitations of a single perspective, and further improving the accuracy of unstructured data processing.

[0122] The specific embodiments of the unstructured data processing device in this application can refer to the examples shown in the above-mentioned unstructured data processing method, and will not be repeated here in this example.

[0123] In one embodiment of the present application, an electronic device for implementing the above-mentioned unstructured data processing method is further provided. The electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-mentioned unstructured data processing method through the computer program.

[0124] Referring to FIG. 4 , FIG. 4 is a schematic diagram illustrating the structure of an electronic device according to an exemplary embodiment of the present application. A computer system 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on programs stored in a read-only memory (ROM) 402 or programs loaded from a storage unit 408 to a random access memory (RAM) 403. RAM 403 also stores various programs and data required for system operation. CPU 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to bus 404.

[0125] The following components are connected to the I / O interface 405: an input section 406 including a keyboard, mouse, and the like; an output section 407 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and speakers; a storage section 408 including devices such as a hard disk; and a communication section 409 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. Removable media 411, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 410 as needed, so that computer programs read from the media can be installed in the storage section 408 as needed.

[0126] In particular, according to embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for executing the methods illustrated in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 409 and / or installed from removable media 411. When executed by the central processing unit (CPU) 401, the computer program performs the various functions defined in the system of the present application.

[0127] It should be noted that the computer-readable medium described in the embodiments of this application may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may, for example, be an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. This propagated data signal may take a variety of forms, including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. A computer program embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0128] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. Among them, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, and the above-mentioned module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0129] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. In some cases, the names of these units do not constitute limitations on the units themselves.

[0130] Another aspect of the present application provides a computer-readable storage medium, which includes a stored computer program. When the computer program is executed, the computer-readable storage medium executes the unstructured data processing method described above. The computer-readable storage medium may be included in the electronic device described in the above embodiments, or may exist independently and not be incorporated into the electronic device.

[0131] Another aspect of the present application provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the unstructured data processing method provided in each of the above embodiments.

[0132] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, any equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical principles disclosed herein are intended to be covered by the claims of the present invention.

Claims

1. A method for processing unstructured data, characterized in that: The method comprises: Acquire unstructured data related to financial risks based on multiple data acquisition channels; Preprocessing the unstructured data to obtain preprocessed unstructured data; Performing feature extraction on the preprocessed unstructured data to obtain target features; Performing uncertainty processing and fuzziness processing on the target feature to obtain uncertainty analysis results and fuzziness processing results; Performing context analysis on the target feature to obtain context-related information; Based on the uncertainty analysis result, the fuzziness processing result and the context association information, the financial risk is evaluated to obtain a financial risk evaluation result.

2. The unstructured data processing method according to claim 1, characterized in that: The multiple data acquisition channels include social media, new network websites, online forums, customer feedback channels, and third-party data suppliers.

3. The unstructured data processing method according to claim 1, characterized in that: The preprocessing of the unstructured data to obtain the preprocessed unstructured data includes: removing noise from the unstructured data to obtain first preprocessed data, detecting missing information in the first preprocessed data, and performing interpolation processing or image restoration processing on the missing information to obtain second preprocessed data; detecting text data in the second preprocessed data to obtain words or sentences, and assigning attention weights to the words or sentences to obtain preprocessed text data; Allocating attention weights on the channel dimension to the image data in the second preprocessed data to obtain preprocessed image data; For the audio data in the second preprocessed data, attention weights are allocated in the frequency domain to obtain preprocessed audio data.

4. The unstructured data processing method according to claim 1, wherein: The extracting features of the pre-processed unstructured data to obtain target features includes: Segmenting text data in the preprocessed unstructured data into words or phrases, removing words or phrases that do not carry information, and converting remaining words into lexical features; Scaling and normalizing the image data of the preprocessed unstructured data, and extracting image features from the image data through a convolutional neural network; Performing noise reduction and equalization processing on the audio data in the preprocessed unstructured data, and extracting audio features in the audio data through Mel-frequency cepstral coefficients; Among them, the vocabulary features, image features, and audio features are included in the target features.

5. The unstructured data processing method according to claim 1, wherein: The target feature is subjected to uncertainty processing and fuzziness processing to obtain uncertainty analysis results and fuzziness processing results, including: Constructing an uncertainty analysis model through a Bayesian network and a membership function, transmitting the target feature to the uncertainty analysis model, and quantifying the output result of the uncertainty analysis model into a confidence interval and credibility to obtain the uncertainty analysis result; Identify the fuzzy expression in the target feature to obtain a fuzzy set, analyze the fuzzy set through fuzzy logic operation to obtain a fuzzy logic operation result, and perform fuzzy reasoning on the fuzzy set based on the fuzzy logic operation result to obtain the fuzziness processing result.

6. The unstructured data processing method according to claim 4, characterized in that: The performing context analysis on the target feature to obtain contextual association information includes: Transmitting the lexical features to a pre-trained sentiment analysis model to obtain sentiment analysis results, wherein the sentiment analysis model is constructed based on a convolutional neural network, and the sentiment analysis results include positive sentiment tendencies, negative sentiment tendencies, and neutral sentiment tendencies; Transmitting the vocabulary features to a pre-trained topic extraction model to obtain key topics and concepts; Performing co-occurrence analysis and dependency analysis on the lexical features to obtain lexical context information corresponding to the key topics and concepts; Transmitting the image features to a pre-trained image recognition model, determining a plurality of objects corresponding to the image features, location information of each of the objects, and scene information, and obtaining image context information based on the plurality of objects, location information, and scene information; The vocabulary context information and the image context information are transmitted to a pre-built joint embedding model to obtain the context association information.

7. The unstructured data processing method according to claim 1, characterized in that: The financial risk is assessed based on the uncertainty analysis result, the fuzziness processing result and the context association information to obtain a financial risk assessment result, including: Acquiring structured data related to the financial risk through the multiple data acquisition channels; fusing the structured data with the uncertainty analysis result, the fuzziness processing result, and the context association information to obtain fused data; The fused data is transmitted to a pre-trained financial risk assessment model to perform financial risk trend assessment to obtain the financial risk assessment result.

8. An unstructured data processing device, characterized in that: The device comprises: A data acquisition module, used to acquire unstructured data related to financial risks based on multiple data acquisition channels; A preprocessing module, configured to preprocess the unstructured data to obtain preprocessed unstructured data; A feature extraction module is used to extract features from the preprocessed unstructured data to obtain target features; An uncertainty processing and fuzziness processing module, used for performing uncertainty processing and fuzziness processing on the target feature to obtain uncertainty analysis results and fuzziness processing results; A context analysis module, configured to perform context analysis on the target feature to obtain contextual association information; The financial risk assessment module is used to assess the financial risk based on the uncertainty analysis result, the fuzziness processing result and the context association information to obtain a financial risk assessment result.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored computer program, wherein the computer program executes the unstructured data processing method according to any one of claims 1 to 7 when executed.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the unstructured data processing method according to any one of claims 1 to 7 through the computer program.

Images