Financial statistical information barrier-free playing method, device, equipment, medium and product

Through the multimodal large language model, the chart data of financial statistics information is converted into descriptive text and played, which solves the problem that visually impaired people cannot obtain information on financial product pages, and achieves a comprehensive understanding of the trend, current situation and prospects of financial products, and improves the Decision-making ability.

CN119090582APending Publication Date: 2024-12-06WEBANK (CHINA)
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411130170.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-16
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The current screen reading function of the device cannot read financial statistics, which limits the ability of visually impaired people to obtain information when browsing financial product pages.

Method used

The multimodal large language model is used to convert the chart data of financial statistics into description text and play it in audio format, including the combination of OCR recognition, image recognition and natural language processing models to generate detailed description text and evaluation text.

Benefits of technology

This enables visually impaired people to effectively obtain chart information on the financial product page, fully understand the product trend, current situation and prospects, make more reasonable decisions, and overcome the limitations of the screen reading function.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119090582A_ABST
    Figure CN119090582A_ABST
Patent Text Reader

Abstract

The invention discloses a barrier-free playing method and device for financial statistical information, equipment, a medium and a product, and relates to the technical field of large models.The barrier-free playing method for the financial statistical information comprises the steps that image data corresponding to the financial statistical information are obtained, and the financial statistical information at least comprises one of a statistical graph or a statistical table; the image data is input into a preset target multi-mode large language model, description texts corresponding to the financial statistical information are output through the target multi-mode large language model, and the target multi-mode large language model is obtained by training multiple sets of financial statistical information and corresponding label texts; and converting the description text into a first audio signal, and playing the first audio signal. According to the technical scheme, financial statistical information can be converted into audio signals to be played, and more comprehensive information is provided for visual impaired people when the visual impaired people select financial products.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of large model technology, and in particular to a method, device, electronic device, computer-readable storage medium and computer program product for barrier-free playback of financial statistical information. Background Art

[0002] Currently, many devices (such as mobile phones) on the market have screen reading functions, which can play the text content on the screen in audio form for the visually impaired, helping them browse text information. However, the current reading function only supports reading text content, not chart information. In some scenarios, such as financial product introduction pages, statistical information such as charts can show more information, helping users to fully understand the trend, current situation and prospects of financial products. Therefore, when the screen reading function of current devices does not support reading financial statistical information, it brings certain limitations to the visually impaired when browsing financial product pages.

[0003] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of this application is to provide a method, device, electronic device, computer-readable storage medium and computer program product for barrier-free playback of financial statistical information, aiming to solve the technical problem that the screen reading function of current devices has limitations for visually impaired people when browsing financial product pages.

[0005] To achieve the above purpose, the present application proposes a method for barrier-free playback of financial statistical information, the method comprising:

[0006] Acquire image data corresponding to financial statistical information, wherein the financial statistical information includes at least one of a statistical graph or a statistical table;

[0007] Inputting the image data into a preset target multimodal large language model, and outputting a description text corresponding to the financial statistical information through the target multimodal large language model, wherein the target multimodal large language model is trained by multiple sets of financial statistical information and corresponding label texts;

[0008] The description text is converted into a first audio signal, and the first audio signal is played.

[0009] In one embodiment, the target multimodal large language model includes an OCR recognition network, an image recognition network, and a natural language processing model;

[0010] The step of inputting the image data into a preset target multimodal large language model and outputting the description text corresponding to the financial statistical information through the target multimodal large language model comprises:

[0011] When the financial statistical information is a statistical graph, after the image data is input into the target multimodal large language model, the statistical graph is subjected to OCR recognition through an OCR recognition network to extract text information from the statistical graph;

[0012] Extracting image features of the statistical graph, inputting the image features into an image recognition network, and identifying and outputting image description information corresponding to the statistical graph through the image recognition network;

[0013] The text information and the image description information are input into a natural language processing model, and processed by the natural language processing model to generate a description text corresponding to the statistical graph.

[0014] In one embodiment, the target multimodal large language model further includes a financial product evaluation model. After the step of generating a description text corresponding to the statistical graph, the method further includes:

[0015] Inputting the description text into a financial product evaluation model, analyzing the description text through the financial product evaluation model, and outputting the evaluation text of the financial product corresponding to the statistical chart, wherein the financial product evaluation model is trained by multiple groups of description texts and corresponding real evaluation labels;

[0016] The evaluation text is converted into a second audio signal, and the second audio signal is played.

[0017] In one embodiment, after the step of outputting the evaluation text of the financial product corresponding to the statistical graph, the method further includes:

[0018] Inputting the description text and the evaluation text into the natural language processing model, summarizing the description text and the evaluation text through the natural language processing model to obtain a summary text;

[0019] The summary text is converted into a third audio signal, and the third audio signal is played.

[0020] In one embodiment, the step of inputting the image into a preset target multimodal large language model and outputting a description text corresponding to the financial statistical information through the target multimodal large language model comprises:

[0021] When the financial statistical information is a statistical table, after the image data is input into the target multimodal large language model, the statistical table is subjected to OCR recognition through an OCR recognition network to obtain text information in the statistical table;

[0022] The text information is input into a natural language processing model, and processed by the natural language processing model to generate a description text corresponding to the statistical table.

[0023] In one embodiment, the method for barrier-free playback of financial statistical information further includes:

[0024] Obtain multiple sets of financial statistical information and corresponding real description texts, and divide them into training set data and test set data;

[0025] Fine-tuning a preset basic multimodal large language model based on the training set data to iteratively update model parameters of the basic multimodal large language model;

[0026] Inputting the financial statistical information in the test set data into the basic multimodal large language model, and outputting corresponding prediction description text through prediction by the basic multimodal large language model;

[0027] Calculating the prediction accuracy of the large language model according to the real description text corresponding to the financial statistical information and the predicted description text;

[0028] When the prediction accuracy is higher than a preset threshold, the fine-tuning process is stopped and the basic multimodal large language model is determined as the target multimodal large language model; otherwise, the process returns to the step of fine-tuning the preset basic multimodal large language model based on the training set data.

[0029] In addition, to achieve the above-mentioned purpose, the present application also proposes a barrier-free playback device for financial statistical information, the barrier-free playback device for financial statistical information comprising:

[0030] A data acquisition module, used to acquire image data corresponding to financial statistical information, wherein the financial statistical information includes at least one of a statistical graph or a statistical table;

[0031] a text generation module, configured to input the image data into a preset target multimodal large language model, and output a description text corresponding to the financial statistical information through the target multimodal large language model, wherein the target multimodal large language model is trained by multiple sets of financial statistical information and corresponding label texts;

[0032] The audio playing module is used to convert the description text into a first audio signal and play the first audio signal.

[0033] In addition, to achieve the above-mentioned purpose, the present application also proposes an electronic device, which includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the computer program is configured to implement the steps of the method for accessible playback of financial statistical information as described above.

[0034] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the method for accessible playback of financial statistical information as described above are implemented.

[0035] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the method for accessible playback of financial statistical information as described above.

[0036] The present application proposes a method for barrier-free playback of financial statistical information. In the method for barrier-free playback of financial statistical information, firstly, image data corresponding to the financial statistical information is obtained, wherein the financial statistical information includes at least one of a statistical chart or a statistical table, and then the image data is input into a preset target multimodal large language model, and the description text corresponding to the financial statistical information is output through the target multimodal large language model, wherein the target multimodal large language model is trained by multiple groups of financial statistical information and corresponding label texts, and finally the description text is converted into a first audio signal, and the first audio signal is played. The technical solution of the present application combines a multimodal large language model to process financial statistical information, and can convert the curve trend, coordinate value and other information in the chart of the financial statistical information into a description text in the form of natural language, and then convert it into an audio signal for playback, so that the visually impaired can effectively obtain the icon information in the financial product page, so that they can fully understand the trend, current situation and prospects of the financial product when selecting the financial product, which is conducive to users making more appropriate decisions, and overcomes the defects of the limitations of the visually impaired when browsing the financial product page through the current screen reading function. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0039] Figure 1 A flowchart diagram of the first embodiment of the method for barrier-free playback of financial statistical information of this application;

[0040] Figure 2 This is a statistical chart of the rate of return of a financial product in the last quarter in the embodiment of the present application;

[0041] Figure 3 A statistical chart showing the performance of a financial product and the performance of the CSI 300 in the past year in the embodiments of the present application;

[0042] Figure 4 This is a schematic diagram of the structure of a device for barrier-free playback of financial statistical information in an embodiment of the present application;

[0043] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the method for accessible playback of financial statistical information in an embodiment of the present application.

[0044] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0045] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0046] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0047] The execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, personal computer, mobile phone, server, etc., or an electronic device, control device, etc. that can realize the above functions.

[0048] The present application embodiment provides a method for barrier-free playback of financial statistical information, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the method for barrier-free playback of financial statistical information of the present application, and the method for barrier-free playback of financial statistical information includes:

[0049] Step S10, acquiring image data corresponding to financial statistical information, wherein the financial statistical information includes at least one of a statistical graph or a statistical table;

[0050] It should be noted that the embodiment of the present application can take a mobile phone as an example to illustrate the implementation of steps S10 to S30. Specifically, when a user (such as a visually impaired person) opens a financial product page while turning on the barrier-free mode of the mobile phone, if the page has financial statistical information (in the form of a statistical chart or a statistical table), and the current screen reading function cannot recognize it, then the image data corresponding to the financial statistical information is obtained (for example, the corresponding image is obtained in the form of a screenshot). These image data can be in a picture format (such as PNG, JPEG) or a more structured format (such as SVG). The image data is used to process and identify the corresponding description text (which may include Chinese characters, numbers, and mathematical symbols, etc.) in combination with a target multimodal large language model with multimodal data (text, image, audio, etc.) processing capability.

[0051] Step S20, inputting the image data into a preset target multimodal large language model, and outputting a description text corresponding to the financial statistical information through the target multimodal large language model, wherein the target multimodal large language model is trained by multiple sets of financial statistical information and corresponding label texts;

[0052] The target multimodal large language model is a pre-set multimodal large language model (such as Chat-Gpt, Tongyi Qianwen, Wenxin Yiyan and other large language models). These large language models have certain image recognition capabilities and can identify the corresponding text description text corresponding to the financial statistical information based on the image data. Moreover, the natural language processing capability of the multimodal large language model is also strong, which can make the description text more in line with the language text used by people in daily life and easier to understand.

[0053] In an embodiment of the present application, when the target multimodal large language model recognizes image data, it will extract detailed information about the financial products in the image, including curve trends (rising or falling, peak values, valley values, etc.), coordinate values, income conditions, net value trends, etc.

[0054] In addition, in order to enable the existing multimodal large language models on the market to have more accurate recognition and understanding capabilities for statistical charts, statistical tables and other information in the financial field, multiple sets of financial statistical information and corresponding label texts are used in advance to fine-tune and train them, so as to obtain a more stable, accurate and more sensitive target multimodal large language model for financial statistical information. The label text is a true and accurate description text written in advance by the staff based on the financial statistical information, which is used as a true label in the model training for supervised learning.

[0055] Step S30: convert the description text into a first audio signal, and play the first audio signal.

[0056] After determining the description text corresponding to the financial statistical information, it can be converted into an audio signal. The conversion process can use mature text-to-audio software or models currently on the market, and the timbre of the first audio signal can also be customized according to user needs.

[0057] Finally, the first audio signal is played through the speaker of the device (such as a mobile phone) so that the visually impaired can obtain the content of the financial statistical information, thereby effectively helping the visually impaired to obtain chart information that the screen reading software cannot directly obtain, so that they can understand more comprehensive information when choosing financial products and make more reasonable decisions.

[0058] In the barrier-free playback method of financial statistical information proposed in the embodiment of the present application, firstly, image data corresponding to the financial statistical information is obtained, wherein the financial statistical information includes at least one of a statistical graph or a statistical table, and then the image data is input into a preset target multimodal large language model, and the description text corresponding to the financial statistical information is output through the target multimodal large language model, wherein the target multimodal large language model is trained by multiple groups of financial statistical information and corresponding label texts, and finally the description text is converted into a first audio signal, and the first audio signal is played. The technical solution of the embodiment of the present application combines a multimodal large language model to process financial statistical information, and can convert the curve trend, coordinate value and other information in the chart of the financial statistical information into a description text in the form of natural language, and then convert it into the form of an audio signal for playback, so that the visually impaired can effectively obtain the icon information in the financial product page, so that they can fully understand the trend, current situation and prospects of the financial product when selecting the financial product, which is conducive to users making more appropriate decisions, and overcomes the defects of the limitations of the visually impaired when browsing the financial product page through the current screen reading function.

[0059] Further, in a feasible implementation, the target multimodal large language model includes an OCR (Optical Character Recognition) recognition network, an image recognition network, and a natural language processing model;

[0060] The step of inputting the image data into a preset target multimodal large language model and outputting the description text corresponding to the financial statistical information through the target multimodal large language model comprises:

[0061] Step S21, when the financial statistical information is a statistical chart, after inputting the image data into the target multimodal large language model, performing OCR recognition on the statistical chart through an OCR recognition network to extract text information in the statistical chart;

[0062] Step S22, extracting image features of the statistical graph, inputting the image features into an image recognition network, and using the image recognition network to recognize and output image description information corresponding to the statistical graph;

[0063] Step S23, inputting the text information and the image description information into a natural language processing model, processing them through the natural language processing model, and generating a description text corresponding to the statistical graph.

[0064] The financial statistical information in the embodiment of the present application may include a statistical chart. It is understood that the statistical chart includes a certain amount of textual information. Such as the title of the statistical chart, the statistical period, the parameter type and unit represented by the horizontal and vertical coordinates, etc., all of which are presented in text form, and the corresponding text information can be recognized by the OCR recognition network. Among them, the OCR network can be a mature image text recognition conversion tool currently on the market, or it can be an OCR recognition model specially used in the embodiment of the present application obtained by training with relevant training data (in text form) in the financial field.

[0065] In addition, the statistical chart also displays information such as the trend of financial products in the form of images. This form of information cannot be recognized by the OCR recognition network, so it is necessary to integrate an image recognition network in the target multimodal large language model to identify elements such as lines and bar charts in the chart to determine the corresponding image description information (in text form). The image recognition network can be a deep learning neural network, such as a convolutional neural network (CNN), which is used to extract key features from the image, such as stock price trend lines, trading volume, price, etc. Exemplarily, when the financial product is a stock, the image recognition network can select and identify features that affect the trend of the stock market, such as opening price, closing price, highest price, lowest price, trading volume, trading volume, technical indicators, such as MACD (moving average convergence difference), RSI (relative strength index), etc.

[0066] It should be noted that in order to improve the recognition accuracy of statistical charts, certain preprocessing can be performed before inputting them into the image recognition network, such as adjusting contrast, resizing, cropping, graying, removing noise, cropping irrelevant parts, and converting them into a format more suitable for processing.

[0067] After obtaining the text information in the statistical graph and the image description information corresponding to the image device, further sorting is required, because these information are scattered and relatively independent, and language organization is required to convert them into a coherent and smooth text. In the embodiment of the present application, a natural language processing model is used to further process the aforementioned text information and image description information. The natural language processing model can combine the input scattered and relatively independent text content with relevant knowledge in the field (such as the financial field) to generate a description text for representing the meaning expressed by the statistical graph.

[0068] For example, Figure 2 As shown in the figure, for the annualized yield statistical chart of a certain financial product, the text information directly expressed in words in the figure includes the title "Annualized yield in the past month", the deadline is "2024-07-11", the yield is "3.17%", and the text information reflected in the horizontal axis is "the time span is from 04-22 to 07-11", and the descriptive information reflected in the statistical chart itself includes the current yield, yield change trend, short-term fluctuations, etc. Among them, "annualized yield in the past month" means that during the period from "04-22 to 07-11", the annualized yield corresponding to each day is the annualized yield within the past month starting from that day.

[0069] For example, for Figure 2 For the yield chart shown, a possible description text (including key data and trends) may include:

[0070] Initial yield: The chart does not show the specific value of the initial yield, but from the trend of the chart we can see that the yield three months ago was around 3.5%.

[0071] Yield change trend: The yield has fluctuated little in the past three months, generally remaining between 2.8% and 3.5%, and the current yield is 3.17%.

[0072] Short-term fluctuations: The chart shows that this product has small fluctuations within three months and the yield is relatively stable.

[0073] Recent Yield: As of July 11, 2024, the yield of this product is 3.17%.

[0074] In the embodiment of the present application, an audio signal playback service corresponding to the descriptive text obtained after identifying the statistical chart of financial products is provided to the visually impaired, so that users can intuitively obtain the current status and changing trends of the financial products, and provide a data basis for users to make reasonable investment decisions.

[0075] Furthermore, in a feasible embodiment, the target multimodal large language model further includes a financial product evaluation model. After the step of generating a description text corresponding to the statistical graph, the method further includes:

[0076] Step S40, inputting the described text into a financial product evaluation model, analyzing the described text through the financial product evaluation model, and outputting the evaluation text of the financial product corresponding to the statistical chart, wherein the financial product evaluation model is trained by multiple groups of described texts and corresponding real evaluation labels;

[0077] Step S50: convert the evaluation text into a second audio signal, and play the second audio signal.

[0078] It should be noted that step S40 is performed after step S20 to generate the evaluation text corresponding to the description text, and step S50 can be performed after step S40 to continue playing the audio signal of the evaluation text after the audio signal of the description text is played.

[0079] It is understandable that for ordinary users, analyzing the pros and cons and situation of a product from the statistical chart and corresponding description text of a financial product requires a certain threshold of professional knowledge. Therefore, in order to provide more comprehensive financial product analysis and evaluation information to the visually impaired, the technical solution of the embodiment of the present application further analyzes the financial product through a pre-set financial product evaluation model, thereby outputting the evaluation text of the financial product and converting it into a corresponding audio signal for playback.

[0080] The financial product evaluation model is trained by multiple groups of description texts and corresponding real evaluation labels, and is used to analyze the characteristics of financial products including overall trends, market environment, investment advice, product stability, etc. based on the description texts. The description texts can be the description information of various financial products in different time periods, including the information that can be directly expressed by the corresponding statistical charts, and the real evaluation labels can be the accurate, rational and comprehensive analysis texts made by experts in the financial field on these description texts, which are used for supervised learning of the model.

[0081] For example, when Figure 2 After the description text corresponding to the yield statistical chart shown is input into the financial product evaluation model, the evaluation text output by the financial product evaluation model may include:

[0082] Overall trend: The yield of this product has changed little in the past three months and remained in a relatively stable range, showing strong stability.

[0083] Market environment: In the past three months, the market environment may have been relatively stable. This product has maintained a stable rate of return in this environment, which shows that it has strong risk resistance.

[0084] Investment advice: Given the stable performance of this product in the past three months, it can be considered as a low-risk investment product. It is particularly suitable for investors who want to obtain stable returns and have a low risk tolerance.

[0085] Product stability: As can be seen from the chart, this product has very little fluctuation in three months, showing extremely high stability.

[0086] In the embodiment of the present application, an audio signal playback service corresponding to the evaluation text obtained after further analysis of the financial product statistical chart is provided for the visually impaired, so that the user can more clearly and intuitively understand the advantages and disadvantages of the financial product, and provide reference suggestions for the user to make reasonable investment decisions.

[0087] In a feasible embodiment, after the step of outputting the evaluation text of the financial product corresponding to the statistical graph, the method further includes:

[0088] Step S60, inputting the description text and the evaluation text into a natural language processing model, summarizing the description text and the evaluation text through the natural language processing model to obtain a summary text;

[0089] Step S70: convert the summary text into a third audio signal, and play the third audio signal.

[0090] It should be noted that step S60 is performed after step S40 to generate a summary text corresponding to both the description text and the evaluation text, and step S70 can be performed after step S50 to continue playing the audio signal of the summary text after the audio signal of the evaluation text is played.

[0091] It is understandable that after the user learns about the description text and evaluation text of the financial product, he or she will have a more comprehensive understanding of the financial product. However, due to the length of the text, the user may not be able to directly determine whether the financial product is worth investing in and whether he or she is suitable for this financial product. Therefore, in order to provide more convenient barrier-free playback services for financial product information to the visually impaired, the technical solution of the embodiment of the present application summarizes the description text and evaluation text through a natural language processing model, thereby outputting the summary text of the financial product and converting it into a corresponding audio signal for playback. Among them, the natural speech processing model can effectively summarize and analyze the existing text content, extract the key points, and finally output a more refined and concise summary text.

[0092] For example, when Figure 2 After the description text and evaluation text corresponding to the yield statistical chart shown are input into the natural language processing model, the summary text output by the natural language processing model may include:

[0093] Based on the above key data and analysis, this product has performed steadily in the past three months, with a yield ranging from 2.8% to 3.5%, and the current yield is 3.17%. The market environment may be relatively stable during this period, and the product has maintained a stable performance in this environment, showing strong risk resistance and stability. For investors who want to obtain stable returns and have a low risk tolerance, this product is a good choice. Investors still need to pay close attention to market dynamics, but overall, this product is a stable low-risk investment option.

[0094] In the embodiment of the present application, an audio signal playback service corresponding to the summary text obtained after summarizing the descriptive text and evaluation text corresponding to the statistical chart of financial products is provided for the visually impaired, so that users can know the advantages and disadvantages of the financial products more comprehensively, and give reasonable investment advice, thereby further improving the user experience.

[0095] In a feasible embodiment, the step of inputting the image into a preset target multimodal large language model and outputting a description text corresponding to the financial statistical information through the target multimodal large language model comprises:

[0096] Step S24, when the financial statistical information is a statistical table, after the image data is input into the target multimodal large language model, the statistical table is subjected to OCR recognition through an OCR recognition network to obtain text information in the statistical table;

[0097] Step S25, inputting the text information into a natural language processing model, processing it through the natural language processing model, and generating a description text corresponding to the statistical table.

[0098] The method for accessible playback of financial statistical information of the embodiment of the present application can also be applied when the financial statistical information is a statistical table, because the statistical table is mainly based on text, and the relationship between the text information in the table is determined by the positional relationship in the table. For example, if the first row of a column is "Date", then the other rows of this column are all date information.

[0099] When the financial statistical information is a statistical table, the text information in the statistical table can be identified by the OCR recognition network in the target multimodal large language model. It should be noted that the text information of the statistical table obtained in step S24 needs to retain the positional relationship between each element (each element refers to the text in an independent cell in the statistical table) to facilitate the natural language processing model to identify and understand the relationship between these text information.

[0100] The natural language processing model can further process the text information in the statistical table to generate a descriptive text with more coherent and fluent sentences that is easy to understand, so as to convert it into a corresponding first audio signal and play it to the visually impaired people, so that the visually impaired people can obtain the information required to be displayed in the statistical table of the financial product page.

[0101] In another feasible embodiment, after obtaining the descriptive text of the statistical table, further analysis and summary can be performed in combination with the financial product evaluation model and the natural language processing model to generate corresponding evaluation text and summary text and play the corresponding audio signal. The specific implementation steps and beneficial effects are the same as steps S40 to S50 and steps S60 to S70, and will not be repeated here.

[0102] The technical solution of the embodiment of the present application can effectively identify financial statistical information of chart type by adopting a multimodal large language model, overcoming the problem that traditional OCR recognition and screen reading functions cannot recognize curves of chart image data, enabling visually impaired people to effectively obtain the information represented by the chart data, thereby promoting the process of financial inclusion.

[0103] For ease of understanding, refer to Figure 3 The performance of a financial product and the performance of the CSI 300 in the past year are shown in the statistical chart. The barrier-free playback method of financial statistical information provided in the embodiment of the present application is displayed when it is recognized that the device (such as a mobile phone) is in barrier-free mode. Figure 3 When the page is displayed ( Figure 3 It shows the yield trend, the rise and fall in the past year, the annual yield of this product is 9.83, and the decline of CSI 300 is 10.28%), obtains the image data of the page, and then inputs the image data into the target multimodal large language model, which outputs the corresponding description text:

[0104] “Initial yield: The chart does not show the specific value of the initial yield, but the trend of the chart shows that the yield was lower one year ago.

[0105] Yield change trend: The yield of this product has generally shown an upward trend in the past year, with an annual yield of 9.83%, while the CSI 300 Index has fallen by 10.28%.

[0106] Short-term fluctuations: The chart shows that this product has experienced some fluctuations within a year, but the overall trend is upward.

[0107] Recent Yield: As of July 15, 2024, the yield of this product is 9.83%. ”

[0108] The description text is then input into the financial product evaluation model, which further analyzes the description text and outputs the corresponding evaluation text:

[0109] “Overall trend: This product has outperformed the CSI 300 Index over the past year, showing strong growth capabilities. The CSI 300 Index performed poorly over the same period, with a 10.28% drop in yield.

[0110] Market environment: In the past year, the market environment may be volatile, resulting in a large decline in the CSI 300 Index. However, this product still performs well in this environment, indicating that it has a certain degree of risk resistance and stability.

[0111] Investment advice: In view of the good performance of this product in the past year, you can consider continuing to hold or increase your investment, especially for investors who hope to obtain stable returns. However, investors still need to pay attention to possible market fluctuations and do a good job of risk management.

[0112] Product stability: As can be seen from the chart, the fluctuation of this product in a year is relatively small, showing a certain degree of stability. "

[0113] The aforementioned description text and analysis text are then input into the natural language processing model, which summarizes them and outputs the corresponding summary text:

[0114] "Based on the above key data and analysis, this product has performed well in the past year, with a yield of 9.83%, significantly better than the CSI 300 Index's -10.28%. Despite the volatile market environment, the product has maintained a stable growth trend, showing strong risk resistance. For investors who want to obtain stable returns, this product is worth considering. However, investors still need to pay close attention to market dynamics and take appropriate risk management measures."

[0115] Finally, the above description text, evaluation text, and summary text are converted into corresponding audio signals and played through the device's speakers for the visually impaired to listen to, providing effective data support and advice guidance for the visually impaired's financial product investment decisions.

[0116] In a feasible embodiment, the method for barrier-free playback of financial statistical information further includes:

[0117] Step A10, obtaining multiple sets of financial statistical information and corresponding real description texts, and dividing them into training set data and test set data;

[0118] Step A20, fine-tuning the preset basic multimodal large language model based on the training set data to iteratively update the model parameters of the basic multimodal large language model;

[0119] Step A30, inputting the financial statistical information in the test set data into the basic multimodal large language model, and outputting the corresponding predicted description text through prediction by the basic multimodal large language model;

[0120] Step A40, calculating the prediction accuracy of the large language model according to the real description text and the predicted description text corresponding to the financial statistical information;

[0121] Step A50, when the prediction accuracy is higher than a preset threshold, stop the fine-tuning process and determine the basic multimodal large language model as the target multimodal large language model; otherwise, return to the step of fine-tuning the preset basic multimodal large language model based on the training set data.

[0122] The embodiment of the present application also provides a method for fine-tuning the basic multimodal large language model to obtain a target multimodal large language model with stronger statistical information recognition ability in the financial field. The basic multimodal large language model (such as Chat-Gpt, Tongyi Qianwen or Wenxin Yiyan, etc.) can effectively improve the performance of the multimodal large language model in this field through financial statistical information in the financial field and the corresponding real description text, and can better effectively identify and understand the chart data in the financial product page in the application, and output a more accurate description text that is closer to natural language.

[0123] During the model training process, it is necessary to obtain a large amount of financial statistical information (including image data of statistical charts or statistical tables) and the corresponding real descriptive text (manually labeled), and then divide each group of financial statistical information into training set data and test set data according to a preset ratio (such as 9:1), which are used for model training and testing respectively.

[0124] Specifically, after the financial statistical information in the training set data is input into the basic multimodal large language model, the basic multimodal large language model will initially identify and understand the financial statistical information, and output the corresponding predicted description text. The current loss function value of the model is calculated according to the real description text and the predicted description text corresponding to the financial statistical information (the loss function value can be calculated by cross entropy loss, KL divergence, etc.), and the model parameters are iteratively updated with the minimization of the loss function value as the optimization goal until the loss function value converges. After the loss function value converges, each group of financial statistical information in the test set data is input into the basic multimodal large language model, and the corresponding predicted description text is predicted and output by the basic multimodal large language model. Then, the prediction accuracy of the large language model is calculated according to the real description text and the predicted description text corresponding to the financial statistical information, wherein the prediction accuracy can be determined by calculating the similarity between the two. When the prediction accuracy is higher than a preset threshold, it can be considered that the basic multimodal large language model meets the application requirements, the fine-tuning process is stopped, and the basic multimodal large language model is determined as the target multimodal large language model for visually impaired people to use to identify the chart information after opening the financial product page.

[0125] The embodiment of the present application provides a method for fine-tuning and training a basic multimodal large language model, so that the multimodal large language model can better recognize and process chart information in the financial field, effectively improve the recognition accuracy, and further improve the reliability of the descriptive text output for financial statistical information and the user experience.

[0126] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the method for accessible playback of financial statistical information of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0127] The present application also provides a barrier-free playback device for financial statistical information, the barrier-free playback device for financial statistical information comprising:

[0128] A data acquisition module 10, used to acquire image data corresponding to financial statistical information, wherein the financial statistical information includes at least one of a statistical graph or a statistical table;

[0129] A text generation module 20, configured to input the image data into a preset target multimodal large language model, and output a description text corresponding to the financial statistical information through the target multimodal large language model, wherein the target multimodal large language model is trained by multiple sets of financial statistical information and corresponding label texts;

[0130] The audio playing module 30 is used to convert the description text into a first audio signal and play the first audio signal.

[0131] In one embodiment, the target multimodal large language model includes an OCR recognition network, an image recognition network, and a natural language processing model;

[0132] The text generation module 20 is also used for:

[0133] When the financial statistical information is a statistical graph, after the image data is input into the target multimodal large language model, the statistical graph is subjected to OCR recognition through an OCR recognition network to extract text information from the statistical graph;

[0134] Extracting image features of the statistical graph, inputting the image features into an image recognition network, and identifying and outputting image description information corresponding to the statistical graph through the image recognition network;

[0135] The text information and the image description information are input into a natural language processing model, and processed by the natural language processing model to generate a description text corresponding to the statistical graph.

[0136] In one embodiment, the target multimodal large language model also includes a financial product evaluation model, and the text generation module 20 is further used to:

[0137] Inputting the description text into a financial product evaluation model, analyzing the description text through the financial product evaluation model, and outputting the evaluation text of the financial product corresponding to the statistical chart, wherein the financial product evaluation model is trained by multiple groups of description texts and corresponding real evaluation labels;

[0138] The audio playback module 30 is also used for:

[0139] The evaluation text is converted into a second audio signal, and the second audio signal is played.

[0140] In one embodiment, the text generation module 20 is further used for:

[0141] Inputting the description text and the evaluation text into the natural language processing model, summarizing the description text and the evaluation text through the natural language processing model to obtain a summary text;

[0142] The audio playback module 30 is also used for:

[0143] The summary text is converted into a third audio signal, and the third audio signal is played.

[0144] In one embodiment, the text generation module 20 is further used for:

[0145] When the financial statistical information is a statistical table, after the image data is input into the target multimodal large language model, the statistical table is subjected to OCR recognition through an OCR recognition network to obtain text information in the statistical table;

[0146] The text information is input into a natural language processing model, and processed by the natural language processing model to generate a description text corresponding to the statistical table.

[0147] In one embodiment, the apparatus for barrier-free playback of financial statistical information further includes a model fine-tuning module, wherein the model fine-tuning module is used to:

[0148] Obtain multiple sets of financial statistical information and corresponding real description texts, and divide them into training set data and test set data;

[0149] Fine-tuning a preset basic multimodal large language model based on the training set data to iteratively update model parameters of the basic multimodal large language model;

[0150] Inputting the financial statistical information in the test set data into the basic multimodal large language model, and outputting corresponding prediction description text through prediction by the basic multimodal large language model;

[0151] Calculating the prediction accuracy of the large language model according to the real description text corresponding to the financial statistical information and the predicted description text;

[0152] When the prediction accuracy is higher than a preset threshold, the fine-tuning process is stopped and the basic multimodal large language model is determined as the target multimodal large language model; otherwise, the process returns to the step of fine-tuning the preset basic multimodal large language model based on the training set data.

[0153] The financial statistical information barrier-free playback device provided in the embodiment of the present application adopts the financial statistical information barrier-free playback method in the above embodiment, which can solve the technical problem that the screen reading function of the current device has limitations for visually impaired people when browsing financial product pages. Compared with the prior art, the beneficial effects of the financial statistical information barrier-free playback device provided in the present application are the same as the beneficial effects of the financial statistical information barrier-free playback method provided in the above embodiment, and the other technical features in the financial statistical information barrier-free playback device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0154] The present application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for accessible playback of financial statistical information in the above-mentioned embodiment 1.

[0155] Reference below Figure 5 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic devices in the embodiments of the present application may include but are not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0156] like Figure 5As shown, the electronic device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the electronic device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows an electronic device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0157] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0158] The electronic device provided by the present application adopts the barrier-free playback method of financial statistical information in the above-mentioned embodiment, which can solve the technical problem that the screen reading function of the current device has limitations for visually impaired people when browsing financial product pages. Compared with the prior art, the beneficial effects of the electronic device provided by the present application are the same as the beneficial effects of the barrier-free playback method of financial statistical information provided by the above-mentioned embodiment, and the other technical features in the electronic device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0159] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0160] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0161] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the method for accessible playback of financial statistical information in the above-mentioned embodiment.

[0162] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0163] The computer-readable storage medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0164] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device executes: obtaining image data corresponding to financial statistical information, wherein the financial statistical information includes at least one of a statistical chart or a statistical table; inputting the image data into a preset target multimodal large language model, and outputting a description text corresponding to the financial statistical information through the target multimodal large language model, wherein the target multimodal large language model is trained by multiple groups of financial statistical information and corresponding label texts; converting the description text into a first audio signal, and playing the first audio signal.

[0165] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0166] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0167] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0168] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned method for accessible playback of financial statistical information, and can solve the technical problem that the screen reading function of the current device has limitations for visually impaired people when browsing financial product pages. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the method for accessible playback of financial statistical information provided by the above-mentioned embodiment, and will not be repeated here.

[0169] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the method for accessible playback of financial statistical information as described above.

[0170] The computer program product provided by this application can solve the technical problem that the screen reading function of current devices is limited for visually impaired people when browsing financial product pages. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the barrier-free playback method of financial statistical information provided by the above embodiment, which will not be repeated here.

[0171] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for barrier-free playback of financial statistical information, It is characterized in that The barrier-free playback method of financial statistical information comprises: Acquire image data corresponding to financial statistical information, wherein the financial statistical information includes at least one of a statistical graph or a statistical table; Inputting the image data into a preset target multimodal large language model, and outputting a description text corresponding to the financial statistical information through the target multimodal large language model, wherein the target multimodal large language model is trained by multiple sets of financial statistical information and corresponding label texts; The description text is converted into a first audio signal, and the first audio signal is played.

2. The method for barrier-free playback of financial statistical information as claimed in claim 1, It is characterized in that The target multimodal large language model includes an OCR recognition network, an image recognition network and a natural language processing model; The step of inputting the image data into a preset target multimodal large language model and outputting the description text corresponding to the financial statistical information through the target multimodal large language model comprises: When the financial statistical information is a statistical graph, after the image data is input into the target multimodal large language model, the statistical graph is subjected to OCR recognition through an OCR recognition network to extract text information from the statistical graph; Extracting image features of the statistical graph, inputting the image features into an image recognition network, and identifying and outputting image description information corresponding to the statistical graph through the image recognition network; The text information and the image description information are input into a natural language processing model, and processed by the natural language processing model to generate a description text corresponding to the statistical graph.

3. The barrier-free playback method of financial statistical information as claimed in claim 2, It is characterized in that The target multimodal large language model also includes a financial product evaluation model. After the step of generating a description text corresponding to the statistical graph, the method further includes: Inputting the description text into a financial product evaluation model, analyzing the description text through the financial product evaluation model, and outputting the evaluation text of the financial product corresponding to the statistical chart, wherein the financial product evaluation model is trained by multiple groups of description texts and corresponding real evaluation labels; The evaluation text is converted into a second audio signal, and the second audio signal is played.

4. The method for barrier-free playback of financial statistical information as claimed in claim 3, It is characterized in that After the step of outputting the evaluation text of the financial product corresponding to the statistical graph, the method further includes: Inputting the description text and the evaluation text into the natural language processing model, summarizing the description text and the evaluation text through the natural language processing model to obtain a summary text; The summary text is converted into a third audio signal, and the third audio signal is played.

5. The barrier-free playback method of financial statistical information as claimed in claim 2, It is characterized in that The step of inputting the image into a preset target multimodal large language model and outputting a description text corresponding to the financial statistical information through the target multimodal large language model comprises: When the financial statistical information is a statistical table, after the image data is input into the target multimodal large language model, the statistical table is subjected to OCR recognition through an OCR recognition network to obtain text information in the statistical table; The text information is input into a natural language processing model, and processed by the natural language processing model to generate a description text corresponding to the statistical table.

6. The method for barrier-free playback of financial statistical information according to any one of claims 1 to 5, It is characterized in that The barrier-free playback method for financial statistical information also includes: Obtain multiple sets of financial statistical information and corresponding real description texts, and divide them into training set data and test set data; Fine-tuning a preset basic multimodal large language model based on the training set data to iteratively update model parameters of the basic multimodal large language model; Inputting the financial statistical information in the test set data into the basic multimodal large language model, and outputting corresponding prediction description text through prediction by the basic multimodal large language model; Calculating the prediction accuracy of the large language model according to the real description text corresponding to the financial statistical information and the predicted description text; When the prediction accuracy is higher than a preset threshold, the fine-tuning process is stopped and the basic multimodal large language model is determined as the target multimodal large language model; otherwise, the process returns to the step of fine-tuning the preset basic multimodal large language model based on the training set data.

7. A barrier-free playback device for financial statistical information, It is characterized in that The barrier-free playback device for financial statistical information comprises: A data acquisition module, used to acquire image data corresponding to financial statistical information, wherein the financial statistical information includes at least one of a statistical graph or a statistical table; a text generation module, configured to input the image data into a preset target multimodal large language model, and output a description text corresponding to the financial statistical information through the target multimodal large language model, wherein the target multimodal large language model is trained by multiple sets of financial statistical information and corresponding label texts; The audio playing module is used to convert the description text into a first audio signal and play the first audio signal.

8. An electronic device, It is characterized in that The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the method for accessible playback of financial statistical information as claimed in any one of claims 1 to 6.

9. A storage medium, It is characterized in that The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the method for accessible playback of financial statistical information as described in any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.