Unstructured data processing method and device, equipment and medium

Abstract

The invention relates to the technical field of data analysis, and provides an unstructured data processing method and device, equipment and a medium, and the method comprises the steps: obtaining unstructured data related to financial risks; pre-processing the unstructured data to obtain pre-processed unstructured data; performing feature extraction on the preprocessed unstructured data to obtain target features; performing uncertainty processing and fuzziness processing on the target features to obtain an uncertainty analysis result and a fuzziness processing result; performing context analysis on the target feature to obtain context associated information; and based on the uncertainty analysis result, the fuzziness processing result and the context association information, evaluating the financial risk to obtain a financial risk evaluation result. In the unstructured data processing method and device, the uncertainty processing and the fuzziness processing can more accurately reflect the actual situation of the financial risk, the context analysis can identify the potential association in the data, and the accuracy of the unstructured data processing is improved.

Description

Technical Field

[0001] The present application relates to the field of data analysis technology, and specifically, to an unstructured data processing method, device, equipment and medium. Background Art

[0002] With the rapid development of financial markets, financial risk control has become particularly important. Traditional financial risk control methods mainly rely on structured data. However, unstructured data contains a wealth of information such as market sentiment and public opinion trends, which are also of great value to financial risk control. Among them, structured data refers to data stored in a predetermined format with a clear logical structure, usually in a table format, containing rows (records) and columns (fields); unstructured data refers to data without a predefined data model or structure, usually containing a large amount of information in the form of text, images, audio, video, etc.

[0003] However, when processing unstructured data, existing technologies have difficulty in obtaining accurate fuzzy logic, implicit information and contextual information due to the complex and changeable limitations of unstructured data related to financial risks, which in turn reduces the accuracy of unstructured data processing.

[0004] Therefore, how to improve the accuracy of unstructured data processing is an urgent problem that needs to be solved. Summary of the invention

[0005] In view of the above problems existing in the prior art, the purpose of this application is to propose an unstructured data processing method, device, equipment and medium to at least solve the technical problem of how to improve the accuracy of unstructured data processing.

[0006] To achieve the above objectives and other related objectives, the present application provides a method for processing unstructured data, the method comprising: Obtain unstructured data related to financial risks based on multiple data acquisition channels; Preprocessing the unstructured data to obtain preprocessed unstructured data; Performing feature extraction on the preprocessed unstructured data to obtain target features; Performing uncertainty processing and fuzziness processing on the target feature to obtain uncertainty analysis results and fuzziness processing results; Performing context analysis on the target feature to obtain context-related information; Based on the uncertainty analysis result, the fuzziness processing result and the context association information, the financial risk is evaluated to obtain a financial risk evaluation result.

[0007] In one embodiment of the present application, the multiple data acquisition channels include social media, new websites, online forums, customer feedback channels, and third-party data suppliers.

[0008] In one embodiment of the present application, the preprocessing of the unstructured data to obtain the preprocessed unstructured data includes: removing noise from the unstructured data to obtain first preprocessed data, detecting missing information in the first preprocessed data, and performing interpolation processing or image restoration processing on the missing information to obtain second preprocessed data; Detecting text data in the second preprocessed data to obtain words or sentences, and assigning attention weights to the words or sentences to obtain preprocessed text data; Allocating attention weights on a channel dimension to the image data in the second preprocessed data to obtain preprocessed image data; For the audio data in the second preprocessed data, attention weights are allocated in the frequency domain to obtain preprocessed audio data.

[0009] In one embodiment of the present application, extracting features from the preprocessed unstructured data to obtain target features includes: Segmenting the text data in the preprocessed unstructured data into words or phrases, removing words or phrases that do not carry information, and converting the remaining words into lexical features; Scaling and normalizing the image data of the preprocessed unstructured data, and extracting image features from the image data through a convolutional neural network; Performing noise reduction and equalization processing on the audio data in the preprocessed unstructured data, and extracting audio features in the audio data through Mel-frequency cepstral coefficients; Among them, the vocabulary features, image features, and audio features are included in the target features.

[0010] In one embodiment of the present application, uncertainty processing and fuzziness processing are performed on the target feature to obtain uncertainty analysis results and fuzziness processing results, including: By using a Bayesian network and a membership function, an uncertainty analysis model is constructed, the target feature is transmitted to the uncertainty analysis model, and the output result of the uncertainty analysis model is quantified into a confidence interval and a credibility, so as to obtain the uncertainty analysis result; Identify the fuzzy expression in the target feature to obtain a fuzzy set, analyze the fuzzy set through fuzzy logic operation to obtain a fuzzy logic operation result, and perform fuzzy reasoning on the fuzzy set based on the fuzzy logic operation result to obtain the fuzziness processing result.

[0011] In one embodiment of the present application, the performing context analysis on the target feature to obtain context association information includes: The lexical features are transmitted to a pre-trained sentiment analysis model to obtain sentiment analysis results, wherein the sentiment analysis model is constructed based on a convolutional neural network, and the sentiment analysis results include positive sentiment tendencies, negative sentiment tendencies, and neutral sentiment tendencies; Transmitting the vocabulary features to a pre-trained topic extraction model to obtain key topics and concepts; Performing co-occurrence analysis and dependency analysis on the lexical features to obtain lexical context information corresponding to the key topics and concepts; Transmitting the image features to a pre-trained image recognition model, determining a plurality of objects corresponding to the image features, location information of each of the objects, and scene information, and obtaining image context information based on the plurality of objects, location information, and scene information; The vocabulary context information and the image context information are transmitted to a pre-built joint embedding model to obtain the context association information.

[0012] In one embodiment of the present application, the financial risk is evaluated based on the uncertainty analysis result, the fuzziness processing result and the context association information to obtain a financial risk evaluation result, including: Acquiring structured data related to the financial risk through the multiple data acquisition channels; Fusing the structured data with the uncertainty analysis result, the fuzziness processing result and the context association information to obtain fused data; The fused data is transmitted to a pre-trained financial risk assessment model to perform financial risk trend assessment to obtain the financial risk assessment result.

[0013] In one embodiment of the present application, an unstructured data processing device is further provided, the device comprising: A data acquisition module, used to acquire unstructured data related to financial risks based on multiple data acquisition channels; A preprocessing module, used for preprocessing the unstructured data to obtain preprocessed unstructured data; A feature extraction module is used to extract features from the preprocessed unstructured data to obtain target features; An uncertainty processing and fuzziness processing module, used to perform uncertainty processing and fuzziness processing on the target feature to obtain uncertainty analysis results and fuzziness processing results; A context analysis module, used to perform context analysis on the target feature to obtain context association information; The financial risk assessment module is used to assess the financial risk based on the uncertainty analysis result, the fuzziness processing result and the context association information to obtain a financial risk assessment result.

[0014] In one embodiment of the present application, a computer-readable storage medium is further provided, wherein the computer-readable storage medium includes a stored computer program, wherein the computer program executes the above-mentioned unstructured data processing method when running.

[0015] In one embodiment of the present application, an electronic device is further provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-mentioned unstructured data processing method through the computer program.

[0016] Beneficial effects of the present invention: First, based on multiple data acquisition channels, unstructured data related to financial risks are acquired; then, the unstructured data are preprocessed to obtain preprocessed unstructured data; then, feature extraction is performed on the preprocessed unstructured data to obtain target features; then, uncertainty processing and fuzziness processing are performed on the target features to obtain uncertainty analysis results and fuzziness processing results; then, context analysis is performed on the target features to obtain context-related information; finally, based on the uncertainty analysis results, the fuzziness processing results and the context-related information, the financial risks are evaluated to obtain financial risk evaluation results. In this application, data is collected through multiple data acquisition channels, which can cover a wider range of information sources and increase the diversity and comprehensiveness of data; preprocessing of unstructured data can eliminate noise, missing values ​​and redundant information in the data and improve data quality; by extracting features from the preprocessed unstructured data, the unstructured raw data can be converted into structured feature vectors, reducing data dimensions and computational complexity. Feature extraction can also highlight key information and improve the data's ability to characterize financial risks; uncertainty processing can quantify the likelihood and impact of risk occurrence, while fuzziness processing can handle fuzzy statements and unclear boundaries in the data. These two processing methods can more accurately reflect the actual situation of financial risks and improve the accuracy of unstructured data processing; through context analysis, potential correlations in the data can be identified, avoiding the limitations of a single perspective and further improving the accuracy of unstructured data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 is a schematic diagram of an application environment of an unstructured data processing method shown in an exemplary embodiment of the present application; Figure 2 is a flowchart of an unstructured data processing method shown in an exemplary embodiment of the present application; Figure 3 is a schematic diagram of an unstructured data processing device shown in an exemplary embodiment of the present application; Figure 4 It is a schematic diagram of the structure of an electronic device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0019] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0020] In one embodiment of the present application, a method for processing unstructured data is provided. Optionally, as an optional implementation, the unstructured data processing method can be applied to, but is not limited to, Figure 1 in the environment shown. Figure 1 is a schematic diagram of an application environment of an unstructured data processing method shown in an exemplary embodiment of the present application, see Figure 1The data processing terminal 101 may be, but is not limited to, a host computer, a desktop computer, a laptop computer, etc. with local computing capabilities. The data processing terminal 101 may, but is not limited to, communicate with the data acquisition terminal 102 and the server 103 through a network. The server 103 may, but is not limited to, perform operations on the database, such as write data operations or read data operations. The above-mentioned data processing terminal 101 may, but is not limited to, include a human-computer interaction screen, a processor, and a memory. The above-mentioned human-computer interaction screen may, but is not limited to, be used to display the results of unstructured data processing. The above-mentioned processor may, but is not limited to, be used to respond to the above-mentioned human-computer interaction operations, perform corresponding operations, or generate corresponding instructions, and send the generated instructions to the server 103. The above-mentioned memory is used to store relevant storage data, such as unstructured data related to financial risks.

[0021] As an optional method, data can be collected through the data collection terminal 102, for example, unstructured data related to financial risks can be collected and preprocessed.

[0022] As an optional manner, the following steps in the unstructured data processing method may be performed on the data processing terminal 101: Obtain unstructured data related to financial risks based on multiple data acquisition channels; Preprocessing the unstructured data to obtain preprocessed unstructured data; Performing feature extraction on the preprocessed unstructured data to obtain target features; Performing uncertainty processing and fuzziness processing on the target feature to obtain uncertainty analysis results and fuzziness processing results; Performing context analysis on the target feature to obtain context-related information; Based on the uncertainty analysis result, the fuzziness processing result and the context association information, the financial risk is evaluated to obtain a financial risk evaluation result.

[0023] As an optional method, the server 103 may obtain the financial risk assessment result transmitted by the data processing terminal 101 and then perform subsequent processing.

[0024] The above method collects data through multiple data acquisition channels, which can cover a wider range of information sources and increase the diversity and comprehensiveness of data; preprocessing unstructured data can eliminate noise, missing values ​​and redundant information in the data and improve data quality; by extracting features from the preprocessed unstructured data, the unstructured raw data can be converted into structured feature vectors, reducing data dimensions and computational complexity. Feature extraction can also highlight key information and improve the data's ability to characterize financial risks; uncertainty processing can quantify the likelihood and impact of risks, while fuzzy processing can handle ambiguous statements and unclear boundaries in the data. These two processing methods can more accurately reflect the actual situation of financial risks and improve the accuracy of unstructured data processing; through context analysis, potential correlations in the data can be identified, avoiding the limitations of a single perspective and further improving the accuracy of unstructured data processing.

[0025] Optionally, in this embodiment, the data processing terminal 101 may be a terminal device configured with a target client, which may include but is not limited to at least one of the following: a laptop, a tablet computer, a handheld computer, a PAD, a desktop computer, etc. The target client may be a client that supports providing unstructured data processing applications, such as a video client, an instant messaging client, a browser client, an education client, etc. The network may include but is not limited to: a wired network, a wireless network, wherein the wired network includes: a local area network, a metropolitan area network and a wide area network, and the wireless network includes: Bluetooth, WIFI and other networks that realize wireless communication. The server may be a single server, or a server cluster consisting of multiple servers, or a cloud server. The above is only an example, and no limitation is made to this in this embodiment.

[0026] As an optional example, this embodiment does not limit the execution subject of the above-mentioned unstructured data processing method. Some or all of the steps of the above-mentioned unstructured data processing method can be executed on the data processing terminal 101. For example, when the data processing terminal 101 is a desktop computer, some or all of the steps of the above-mentioned unstructured data processing method can be executed on the desktop computer.

[0027] In one embodiment of the present application, a method for processing unstructured data is provided. Figure 2 is a flowchart of an unstructured data processing method shown in an exemplary embodiment of the present application, see Figure 2 The unstructured data processing method includes the following steps S210 to S260: In step S210, unstructured data related to financial risks is acquired based on multiple data acquisition channels.

[0028] Among them, unstructured data related to financial risks are collected from multiple data acquisition channels, including social media, news reports, market reports, information released by the government and regulatory agencies, internal reports and communication records.

[0029] In step S220, the unstructured data is preprocessed to obtain preprocessed unstructured data.

[0030] Among them, preprocessing unstructured data is one of the key steps in data analysis, the purpose of which is to improve data quality and make the data more suitable for subsequent feature extraction and modeling. Preprocessing text data in unstructured data can include data cleaning and standardization to obtain clean, segmented, standardized text, as well as word vector representation or context-sensitive word vector representation; preprocessing image data in unstructured data can include image standardization and data enhancement, and can also segment the image into different regions to extract specific areas of interest, remove background information in the image, retain the main part, and use edge detection algorithms to enhance the edge information of the image, and finally obtain images of uniform size and color standardization, as well as data-enhanced image collections; preprocessing audio data in unstructured data can include audio data standardization and data enhancement, and can also use filters to remove background noise in the audio, and can also divide the audio signal into short time windows for subsequent feature extraction, and finally obtain audio with uniform sampling rate and volume standardization, as well as data-enhanced audio collections.

[0031] In step S230, feature extraction is performed on the preprocessed unstructured data to obtain target features.

[0032] Among them, feature extraction of preprocessed unstructured data is a key step in data analysis and modeling. The goal of feature extraction is to extract representations from raw data that can effectively reflect the essential characteristics of the data. Target features can include text features, image features, and audio features. Text data features can include word vectors, context-sensitive word vectors, sentence vectors, topic distributions, and sentiment scores; image features can include manual features, deep learning features, object detection results, and image segmentation results; audio features can include time domain features, frequency domain features, time-frequency features, and deep learning features.

[0033] In step S240, uncertainty processing and fuzziness processing are performed on the target feature to obtain uncertainty analysis results and fuzziness processing results.

[0034] Among them, uncertainty processing and fuzziness processing of target features are to better manage and express uncertainty and fuzziness in unstructured data. The uncertainty analysis results and fuzziness processing results can be used to more accurately assess financial risks and help decision makers better understand and deal with uncertainty factors. In addition, the uncertainty analysis results and fuzziness processing results can also be used to build more robust prediction models and improve the effectiveness of financial risk management.

[0035] In step S250, context analysis is performed on the target feature to obtain context association information.

[0036] Among them, contextual analysis of target features is an important step to understand the relationship between data and to mine potential patterns and insights. Contextual information refers to the association and dependency between features and between features and external environments (such as time, location, user behavior, etc.). This kind of analysis is particularly important for improving the accuracy of financial risk assessment because it can reveal the complex factors and dynamic changes behind financial risks.

[0037] In step S260, the financial risk is evaluated based on the uncertainty analysis result, the fuzziness processing result and the context association information to obtain a financial risk evaluation result.

[0038] For example, for credit risk assessment, we can first assess the uncertainty of the borrower's repayment ability, income level and other characteristics, then process the ambiguous information in the application form, such as income range, years of work, etc., and then analyze the borrower's financial report, credit history, social media comments, etc., extract keywords, identify key entities and relationships, and finally use the above information to train the credit risk assessment model to generate the borrower's risk score.

[0039] For example, for market risk monitoring, one can first monitor news reports, social media posts, etc. to assess the uncertainty of market fluctuations, then process the ambiguous information in the news reports, such as market expectations, expert forecasts, etc., and then analyze the entity relationships and emotional tendencies in the news reports to assess market sentiment. Finally, use the above information to train the market risk monitoring model to generate a market risk score.

[0040] For example, for operational risk management, you can first assess the uncertainty of operational risk events, such as system failures, human errors, etc., then process the ambiguous information in internal reports, such as event descriptions, impact levels, etc., and then analyze the key entities and event relationships in the internal reports to assess operational risks. Finally, use the above information to train the operational risk management model to generate a risk score for operational risks.

[0041] By adopting the above-mentioned embodiments provided by the present application, data is collected through multiple data acquisition channels, which can cover a wider range of information sources and increase the diversity and comprehensiveness of the data; preprocessing the unstructured data can eliminate noise, missing values ​​and redundant information in the data and improve the data quality; by extracting features from the preprocessed unstructured data, the unstructured raw data can be converted into structured feature vectors, reducing the data dimension and computational complexity. Feature extraction can also highlight key information and improve the data's ability to characterize financial risks; uncertainty processing can quantify the possibility and impact of risks, while fuzzy processing can handle fuzzy expressions and unclear boundaries in the data. These two processing methods can more accurately reflect the actual situation of financial risks and improve the accuracy of unstructured data processing; through context analysis, potential correlations in the data can be identified, avoiding the limitations of a single perspective and further improving the accuracy of unstructured data processing.

[0042] In one embodiment of the present application, the multiple data acquisition channels include social media, new network websites, online forums, customer feedback channels, and third-party data suppliers.

[0043] In one embodiment of the present application, the preprocessing of the unstructured data to obtain the preprocessed unstructured data includes: removing noise from the unstructured data to obtain first preprocessed data, detecting missing information in the first preprocessed data, and performing interpolation processing or image restoration processing on the missing information to obtain second preprocessed data; Detecting text data in the second preprocessed data to obtain words or sentences, and assigning attention weights to the words or sentences to obtain preprocessed text data; Allocating attention weights on a channel dimension to the image data in the second preprocessed data to obtain preprocessed image data; For the audio data in the second preprocessed data, attention weights are allocated in the frequency domain to obtain preprocessed audio data.

[0044] In this embodiment, by removing the noise in the unstructured data, the purity of the data can be significantly improved, the interference factors in the subsequent processing can be reduced, and the accuracy of feature extraction and modeling can be improved. Interpolation or image restoration of missing information can make up for the gaps in the data, making the data more complete, thereby improving the effect of model training. By assigning attention weights to vocabulary, sentences, image channels, and audio frequency domains, important information in the data can be enhanced, unnecessary details can be reduced, and the model's ability to capture key information can be improved. The attention mechanism can make the model more focused on the important parts of the input data, thereby improving the learning efficiency and generalization ability of the model.

[0045] In one embodiment of the present application, the feature extraction of the preprocessed unstructured data to obtain target features includes: Segment the text data in the preprocessed unstructured data into words or phrases, remove the words or phrases that do not carry information, and convert the remaining vocabulary into vocabulary features; Perform scaling and normalization processing on the image data of the preprocessed unstructured data, and extract image features in the image data through a convolutional neural network; Perform noise reduction and equalization processing on the audio data in the preprocessed unstructured data, and extract audio features in the audio data through Mel-frequency cepstral coefficients; Among them, the vocabulary features, image features, and audio features are included in the target features.

[0046] In this embodiment, segmenting the text data in the preprocessed unstructured data into words or phrases can decompose the text data into manageable units, laying a foundation for subsequent analysis. Removing the words or phrases that do not carry information, such as removing stop words (such as "de", "he", etc.), can reduce the noise of the data and improve the efficiency and effect of subsequent processing. Converting the remaining vocabulary into features can quantify the text data and make it available for machine learning models. Performing scaling and normalization processing on the image data of the preprocessed unstructured data ensures that the image data has a unified size and ratio, facilitating model processing. At the same time, normalization can accelerate the convergence speed of model training. By removing background noise and adjusting the volume, the quality of the audio data can be improved, making the audio features more prominent and clear.

[0047] In one embodiment of the present application, the uncertainty processing and ambiguity processing of the target features to obtain uncertainty analysis results and ambiguity processing results include: Construct an uncertainty analysis model through a Bayesian network and a membership function, transmit the target features to the uncertainty analysis model, and quantify the output result of the uncertainty analysis model into a confidence interval and a credibility to obtain the uncertainty analysis result; Identify the fuzzy expressions in the target features to obtain a fuzzy set, analyze the fuzzy set through fuzzy logic operations to obtain a fuzzy logic operation result, and perform fuzzy reasoning on the fuzzy set based on the fuzzy logic operation result to obtain the ambiguity processing result.

[0048] In this embodiment, the uncertainty analysis model is constructed through the Bayesian network, which can quantify the uncertainty of the target feature and present it in the form of confidence intervals and credibility. This helps decision makers to more accurately assess risks, predict future trends, and make more robust decisions accordingly. Bayesian networks can handle multi-source information fusion and uncertainty propagation, and even if the input data is noisy or inaccurate, relatively reliable results can be obtained through probabilistic reasoning. This ability makes the model more robust in the face of complex and changing financial environments. The quantitative representation of confidence intervals and credibility enables risk managers to more clearly understand the scope and possibility of risks, thereby formulating more refined risk management strategies.

[0049] In this embodiment, in financial data, there are often fuzzy expressions (such as "high", "low", "medium", etc.), and these fuzzy information are difficult to express with traditional precise numerical values. By identifying fuzzy expressions and constructing fuzzy sets, fuzzy logic can process these fuzzy information, thereby retaining the integrity and richness of the data. Fuzzy logic operations and fuzzy reasoning can simulate the fuzzy thinking process of humans and analyze and reason about fuzzy sets. This capability enables the system to handle more complex and uncertain financial problems, such as evaluating market trends, predicting customer needs, etc. Fuzzy processing allows the system to make flexible decisions within a certain range, rather than relying solely on precise numerical results. This flexibility enables the system to better adapt to the rapid changes and uncertainties in the financial market.

[0050] In one embodiment of the present application, the performing context analysis on the target feature to obtain context association information includes: The lexical features are transmitted to a pre-trained sentiment analysis model to obtain sentiment analysis results, wherein the sentiment analysis model is constructed based on a convolutional neural network, and the sentiment analysis results include positive sentiment tendencies, negative sentiment tendencies, and neutral sentiment tendencies; Transmitting the vocabulary features to a pre-trained topic extraction model to obtain key topics and concepts; Performing co-occurrence analysis and dependency analysis on the lexical features to obtain lexical context information corresponding to the key topics and concepts; Transmitting the image features to a pre-trained image recognition model, determining a plurality of objects corresponding to the image features, location information of each of the objects, and scene information, and obtaining image context information based on the plurality of objects, location information, and scene information; The vocabulary context information and the image context information are transmitted to a pre-built joint embedding model to obtain the context association information.

[0051] In this embodiment, by processing the vocabulary features through the sentiment analysis model (based on the convolutional neural network), the positive sentiment tendency, negative sentiment tendency and neutral sentiment tendency in the text can be identified. This helps to understand the emotional color in the text and provides a basis for subsequent sentiment analysis and sentiment evaluation. By processing the vocabulary features through the pre-trained topic extraction model, the key topics and concepts in the text can be extracted. This helps to understand the core content of the text and provides support for subsequent topic analysis and information extraction. Through co-occurrence analysis and dependency analysis, the association information between words can be captured, which is crucial for understanding the contextual relationship in the text. Co-occurrence analysis can reveal which words often appear together, while dependency analysis can reveal the grammatical relationship between words. Co-occurrence analysis and dependency analysis can enhance the model's ability to understand the text, especially when processing complex financial texts, it can better understand the logical and semantic relationships between words. Through the pre-trained image recognition model, multiple objects in the image and their location information can be identified, which is very important for understanding the scene and content in the image. The image recognition model can not only identify objects, but also extract scene information. For example, in financial risk assessment, financial-related scenes (such as banks, exchanges, etc.) can be identified. By fusing lexical context information and image context information through a joint embedding model, we can achieve the complementarity of text and image information, thereby obtaining more comprehensive contextual association information.

[0052] In one embodiment of the present application, the financial risk is evaluated based on the uncertainty analysis result, the fuzziness processing result and the context association information to obtain a financial risk evaluation result, including: Acquiring structured data related to the financial risk through the multiple data acquisition channels; Fusing the structured data with the uncertainty analysis result, the fuzziness processing result and the context association information to obtain fused data; The fused data is transmitted to a pre-trained financial risk assessment model to perform financial risk trend assessment to obtain the financial risk assessment result.

[0053] It can be seen from the above embodiments that collecting data through multiple data acquisition channels can cover a wider range of information sources and increase the diversity and comprehensiveness of data; preprocessing unstructured data can eliminate noise, missing values ​​and redundant information in the data and improve data quality; by extracting features from the preprocessed unstructured data, the unstructured raw data can be converted into structured feature vectors, reducing data dimensions and computational complexity. Feature extraction can also highlight key information and improve the data's ability to characterize financial risks; uncertainty processing can quantify the likelihood and impact of risks, while fuzzy processing can handle fuzzy expressions and unclear boundaries in the data. These two processing methods can more accurately reflect the actual situation of financial risks and improve the accuracy of unstructured data processing; through context analysis, potential correlations in the data can be identified, avoiding the limitations of a single perspective and further improving the accuracy of unstructured data processing.

[0054] In one embodiment of the present application, an unstructured data processing device is also provided. Figure 3 is a schematic diagram of an unstructured data processing device shown in an exemplary embodiment of the present application, see Figure 3 , the device comprises: A data acquisition module 301 is used to acquire unstructured data related to financial risks based on multiple data acquisition channels; A preprocessing module 302 is used to preprocess the unstructured data to obtain preprocessed unstructured data; A feature extraction module 303 is used to extract features from the preprocessed unstructured data to obtain target features; The uncertainty processing and fuzziness processing module 304 is used to perform uncertainty processing and fuzziness processing on the target feature to obtain uncertainty analysis results and fuzziness processing results; A context analysis module 305 is used to perform context analysis on the target feature to obtain context association information; The financial risk assessment module 306 is used to assess the financial risk based on the uncertainty analysis result, the fuzziness processing result and the context association information to obtain a financial risk assessment result.

[0055] The unstructured data processing device in the embodiment of the present application collects data through multiple data acquisition channels, which can cover a wider range of information sources and increase the diversity and comprehensiveness of data; preprocessing the unstructured data can eliminate noise, missing values ​​and redundant information in the data and improve the data quality; by extracting features from the preprocessed unstructured data, the unstructured raw data can be converted into structured feature vectors, reducing data dimensions and computational complexity. Feature extraction can also highlight key information and improve the data's ability to characterize financial risks; uncertainty processing can quantify the likelihood and impact of risks, while fuzzy processing can handle fuzzy expressions and unclear boundaries in the data. These two processing methods can more accurately reflect the actual situation of financial risks and improve the accuracy of unstructured data processing; through context analysis, potential correlations in the data can be identified, avoiding the limitations of a single perspective and further improving the accuracy of unstructured data processing.

[0056] The specific embodiments of the unstructured data processing device in the present application can refer to the examples shown in the above-mentioned unstructured data processing method, which will not be repeated here.

[0057] In one embodiment of the present application, an electronic device for implementing the above-mentioned unstructured data processing method is also provided. The electronic device includes a memory and a processor, the memory stores a computer program, and the processor is configured to execute the above-mentioned unstructured data processing method through the computer program.

[0058] See also Figure 4 , Figure 4 4 is a schematic diagram of the structure of an electronic device shown in an exemplary embodiment of the present application. The computer system 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 402 or the program loaded from the storage part 408 to the random access memory (RAM) 403, such as executing the method described in the above embodiment. In RAM 403, various programs and data required for system operation are also stored. CPU 401, ROM 402 and RAM 403 are connected to each other through bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0059] The following components are connected to the I / O interface 405: an input section 406 including a keyboard, a mouse, etc.; an output section 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 410 as needed so that a computer program read therefrom is installed into the storage section 408 as needed.

[0060] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 409, and / or installed from a removable medium 411. When the computer program is executed by the central processing unit (CPU) 401, various functions defined in the system of the present application are executed.

[0061] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier, which carries a computer-readable computer program. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. A computer program contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0062] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0063] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not constitute limitations on the units themselves in some cases.

[0064] Another aspect of the present application further provides a computer-readable storage medium, the computer-readable storage medium including a stored computer program, wherein the computer program executes the above-mentioned unstructured data processing method when it is run. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0065] Another aspect of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the unstructured data processing method provided in each of the above embodiments.

[0066] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present invention. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A method for processing unstructured data, characterized in that: The method comprises: Obtain unstructured data related to financial risks based on multiple data acquisition channels; Preprocessing the unstructured data to obtain preprocessed unstructured data; Performing feature extraction on the preprocessed unstructured data to obtain target features; Performing uncertainty processing and fuzziness processing on the target feature to obtain uncertainty analysis results and fuzziness processing results; Performing context analysis on the target feature to obtain context-related information; Based on the uncertainty analysis result, the fuzziness processing result and the context association information, the financial risk is evaluated to obtain a financial risk evaluation result.

2. The unstructured data processing method according to claim 1, characterized in that: The multiple data acquisition channels include social media, new network websites, online forums, customer feedback channels, and third-party data suppliers.

3. The unstructured data processing method according to claim 1, characterized in that: The preprocessing of the unstructured data to obtain the preprocessed unstructured data includes: removing noise from the unstructured data to obtain first preprocessed data, detecting missing information in the first preprocessed data, and performing interpolation processing or image restoration processing on the missing information to obtain second preprocessed data; Detecting text data in the second preprocessed data to obtain words or sentences, and assigning attention weights to the words or sentences to obtain preprocessed text data; Allocating attention weights on a channel dimension to the image data in the second preprocessed data to obtain preprocessed image data; For the audio data in the second preprocessed data, attention weights are allocated in the frequency domain to obtain preprocessed audio data.

4. The unstructured data processing method according to claim 1, characterized in that: The extracting features of the preprocessed unstructured data to obtain target features includes: Segmenting the text data in the preprocessed unstructured data into words or phrases, removing words or phrases that do not carry information, and converting the remaining words into lexical features; Scaling and normalizing the image data of the preprocessed unstructured data, and extracting image features from the image data through a convolutional neural network; Performing noise reduction and equalization processing on the audio data in the preprocessed unstructured data, and extracting audio features in the audio data through Mel-frequency cepstral coefficients; Among them, the vocabulary features, image features, and audio features are included in the target features.

5. The unstructured data processing method according to claim 1, characterized in that: The target feature is subjected to uncertainty processing and fuzziness processing to obtain uncertainty analysis results and fuzziness processing results, including: By using a Bayesian network and a membership function, an uncertainty analysis model is constructed, the target feature is transmitted to the uncertainty analysis model, and the output result of the uncertainty analysis model is quantified into a confidence interval and a credibility, so as to obtain the uncertainty analysis result; Identify the fuzzy expression in the target feature to obtain a fuzzy set, analyze the fuzzy set through fuzzy logic operation to obtain a fuzzy logic operation result, and perform fuzzy reasoning on the fuzzy set based on the fuzzy logic operation result to obtain the fuzziness processing result.

6. The unstructured data processing method according to claim 4, characterized in that: The performing context analysis on the target feature to obtain context association information includes: The lexical features are transmitted to a pre-trained sentiment analysis model to obtain sentiment analysis results, wherein the sentiment analysis model is constructed based on a convolutional neural network, and the sentiment analysis results include positive sentiment tendencies, negative sentiment tendencies, and neutral sentiment tendencies; Transmitting the vocabulary features to a pre-trained topic extraction model to obtain key topics and concepts; Performing co-occurrence analysis and dependency analysis on the lexical features to obtain lexical context information corresponding to the key topics and concepts; Transmitting the image features to a pre-trained image recognition model, determining a plurality of objects corresponding to the image features, location information of each of the objects, and scene information, and obtaining image context information based on the plurality of objects, location information, and scene information; The vocabulary context information and the image context information are transmitted to a pre-built joint embedding model to obtain the context association information.

7. The unstructured data processing method according to claim 1, characterized in that: The step of evaluating the financial risk based on the uncertainty analysis result, the fuzziness processing result and the context association information to obtain a financial risk evaluation result includes: Acquiring structured data related to the financial risk through the multiple data acquisition channels; Fusing the structured data with the uncertainty analysis result, the fuzziness processing result and the context association information to obtain fused data; The fused data is transmitted to a pre-trained financial risk assessment model to perform financial risk trend assessment to obtain the financial risk assessment result.

8. An unstructured data processing device, characterized in that: The device comprises: A data acquisition module, used to acquire unstructured data related to financial risks based on multiple data acquisition channels; A preprocessing module, used for preprocessing the unstructured data to obtain preprocessed unstructured data; A feature extraction module is used to extract features from the preprocessed unstructured data to obtain target features; An uncertainty processing and fuzziness processing module, used to perform uncertainty processing and fuzziness processing on the target feature to obtain uncertainty analysis results and fuzziness processing results; A context analysis module, used to perform context analysis on the target feature to obtain context association information; The financial risk assessment module is used to assess the financial risk based on the uncertainty analysis result, the fuzziness processing result and the context association information to obtain a financial risk assessment result.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises a stored computer program, wherein the computer program executes the unstructured data processing method according to any one of claims 1 to 7 when executed.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the unstructured data processing method according to any one of claims 1 to 7 through the computer program.

Images