Automatic data cleaning system based on big data service

CN118964339AInactive Publication Date: 2024-11-15SHANGHAI YULINGSHU NETWORK TECH CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202410966185.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-15
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The prior art is difficult to achieve full automation in the process of voice-to-text conversion, resulting in the need of manual screening, affecting the convenience and accuracy of data cleaning.

Method used

An automated data cleaning system based on big data services was designed, including data processing module, database and cleaning module. Through voice input submodule, text conversion submodule, recording submodule and cleaning submodule, we can identify and correct voice to text. The errors of learning user habits and optimizing the data cleaning process to reduce manual operations.

Benefits of technology

It realizes automatic identification and correction of errors during the speech to text process, improves the accuracy of language to text, reduces manual screening operations, and improves the automation of data cleaning.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118964339A_ABST
    Figure CN118964339A_ABST
Patent Text Reader

Abstract

The invention discloses an automatic data cleaning system based on big data service. The automatic data cleaning system comprises an automatic data cleaning system and a display terminal, wherein the automatic data cleaning system comprises a data processing module, a database and a cleaning module; the data processing module is used for receiving voice of a user and converting the voice into characters; the database is used for storing data; the cleaning module is used for cleaning wrong places in voice-to-text conversion to ensure the accuracy of voice-to-text conversion of a user, the data processing module comprises a voice input sub-module, a text conversion sub-module, a recording sub-module and a prompting sub-module, the voice input sub-module is used for inputting the voice of the user, and the text conversion sub-module is used for converting the voice of the user into the text. The character conversion sub-module is used for converting voice of a user into characters, the recording sub-module is used for recording the number of wrong times of character conversion, and the automatic data cleaning function is conveniently and efficiently achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention is applied to the technical field of automated data cleaning, and is named as an automated data cleaning system based on big data services. Background Art

[0002] Big data generally refers to a collection of data that cannot be captured, managed, and processed using conventional software tools within a certain period of time. It is also a concept that has received extensive attention, discussion, and research in recent years. Accordingly, big data processing technology that can quickly obtain valuable information from various types of big data has also been developed in recent years.

[0003] Data cleaning is an important part of the application of big data processing technology, which detects and corrects identifiable errors in data files.

[0004] Now with the development of technology and the Internet, a speech-to-text function has been developed to facilitate people's text input. However, this function is difficult to distinguish when ambiguous words appear during speech-to-text conversion and requires manual screening. It cannot achieve fully automated functions, so it is inconvenient to use.

[0005] Therefore, it is necessary to provide an automated data cleaning system based on big data services, which can reduce manual screening operations and realize automated data cleaning functions. Summary of the invention

[0006] The purpose of the present invention is to provide an automated data cleaning system based on big data services to solve the problems raised in the above background technology.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions: an automated data cleaning system based on big data services, comprising an automated data cleaning system and a display terminal: the automated data cleaning system comprises a data processing module, a database and a cleaning module;

[0008] The data processing module is used to receive the user's voice and convert the voice into text;

[0009] The database is used to store data;

[0010] The cleaning module is used to clean up the errors in the speech-to-text conversion to ensure the accuracy of the user's speech-to-text conversion.

[0011] In one embodiment, the data processing module includes a voice input submodule, a text conversion submodule, a recording submodule and a prompt submodule, wherein the voice input submodule is used to input the user's voice, the text conversion submodule is used to convert the user's voice into text, the recording submodule is used to count the number of errors in text conversion, and the prompt submodule is used to give prompts to the user;

[0012] The voice input submodule includes a noise filtering unit to filter the surrounding noisy sounds;

[0013] The text conversion submodule includes a recognition unit for recognizing whether there is an error in the speech-converted text.

[0014] The voice input submodule and the text conversion submodule are electrically connected to the display terminal.

[0015] In one embodiment, the database includes an analysis submodule, a data update submodule and a memory submodule, wherein the analysis submodule is electrically connected to the recording submodule and is used to analyze the error rate of speech-to-text conversion, the data update submodule is used to update the error situation of speech-to-text conversion to the database, and the recording submodule is used to record the user's usage habits;

[0016] The memory submodule includes a learning unit and an extracting unit, which are used to learn the user's usage habits.

[0017] In one embodiment, the cleaning module includes a cleaning submodule and a manual operation submodule. The cleaning submodule is used for preliminary cleaning of data, and the manual operation submodule is used for manual operation screening.

[0018] In one embodiment, the database further includes a period statistics submodule and a clearing submodule. The period statistics submodule is used to count the period of data in the database, and the clearing submodule is used to delete infrequently used data to reserve more memory for the database.

[0019] In one embodiment, the automated data cleaning system includes the following specific operating steps:

[0020] S1. The voice of the user is recorded through the voice input submodule, and then the voice of the user is converted into text through the text conversion submodule, and the text is displayed on the display terminal. When the voice input submodule is used, the noise filtering unit is activated to filter out external noise;

[0021] S2, when the voice input submodule records the user's voice, the recording submodule records the user's language habits and transmits them to the memory submodule;

[0022] S3. When the speech is converted into text, the recognition unit identifies whether the converted text has errors. If the recognized text has errors, the text is cleaned up.

[0023] S4, recording the number of errors through the recording submodule, and transmitting the signal to the analysis submodule, determining the error rate of the user's speech-to-text conversion, and analyzing the quality of the speech-to-text conversion in combination with the external noise level;

[0024] S5. In order to ensure the memory of the database, delete useless data.

[0025] Compared with the prior art, the beneficial effect achieved by the present invention is as follows: the present invention is provided with an automatic data cleaning system, which can identify the user's language habits and external noise when converting speech to text, extract relevant data, perform data cleaning on erroneous text, and correct the errors in speech-to-text conversion without the need for manual operation each time, thereby realizing the data cleaning function. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The technical solution and other beneficial effects of the present application will be made apparent by describing in detail the specific implementation methods of the present application in conjunction with the accompanying drawings.

[0027] In the attached picture:

[0028] Figure 1 is a schematic diagram of the automated cleaning data system of the present invention;

[0029] Figure 2 It is a schematic diagram of a database of the present invention; DETAILED DESCRIPTION

[0030] The disclosure below provides many different embodiments or examples to realize the different structures of the present application. In order to simplify the disclosure of the present application, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present application. In addition, the present application can repeat reference numbers and / or reference letters in different examples, and this repetition is for the purpose of simplification and clarity, which itself does not indicate the relationship between the various embodiments and / or settings discussed. In addition, the various specific processes and material examples provided by the present application, but those of ordinary skill in the art can appreciate the application of other processes and / or the use of other materials.

[0031] See also Figure 1-2 ,The present invention provides a technical solution: an automated data cleaning system based on big data services, comprising 1. an automated data cleaning system based on big data services, comprising an automated data cleaning system and a display terminal,: the automated data cleaning system comprises a data processing module, a database and a cleaning module;

[0032] The data processing module is used to receive the user's voice and convert the voice into text;

[0033] The database is used to store data, which consists of various text segments, words, and various alphabetic punctuation marks;

[0034] The cleaning module is used to remove errors in the speech-to-text conversion to ensure the accuracy of the user's speech-to-text conversion.

[0035] The data processing module includes a voice input submodule, a text conversion submodule, a recording submodule and a prompt submodule. The voice input submodule is used to input the user's voice, the text conversion submodule is used to convert the user's voice into text, the recording submodule is used to count the number of errors in text conversion, and the prompt submodule is used to give prompts to the user.

[0036] The voice input submodule includes a noise filtering unit to filter out the surrounding noisy sounds;

[0037] The text conversion submodule includes a recognition unit to identify whether there are errors in the speech-to-text conversion.

[0038] The voice input submodule and the text conversion submodule are electrically connected to the display terminal.

[0039] It is necessary to explain whether the positioning is wrong, including whether the voice-to-text conversion is fuzzy, repeated and other problems.

[0040] The database includes an analysis submodule, a data update submodule and a memory submodule. The analysis submodule is electrically connected to the recording submodule and is used to analyze the error rate of speech-to-text conversion. The data update submodule is used to update the error situation of speech-to-text conversion to the database. The recording submodule is used to record the user's usage habits.

[0041] The memory submodule includes a learning unit and an extraction unit, which are used to learn the user's usage habits and can automatically obtain the content of the database to automatically generate text. The extraction unit is used to extract text segments in the data.

[0042] The cleaning module includes a cleaning submodule and a manual operation submodule. The cleaning submodule is used for preliminary cleaning of data, and the manual operation submodule is used for manual operation screening.

[0043] The database also includes a cycle statistics submodule and a clearing submodule. The cycle statistics submodule is used to count the cycles of data in the database, and the clearing submodule is used to delete infrequently used data to reserve more memory for the database.

[0044] The automated data cleaning system includes the following specific steps:

[0045] S1. The voice of the user is recorded through the voice input submodule, and then the voice of the user is converted into text through the text conversion submodule, and the text is displayed on the display terminal. When the voice input submodule is used, the noise filtering unit is activated to filter out external noise and purify the user's voice, so as to obtain more accurate information;

[0046] S2, when the voice input submodule records the user's voice, the recording submodule records the user's language habits and transmits them to the memory submodule;

[0047] Specifically, since each user has different pause and speaking habits, when the user speaks, the user's language habits are recorded by the memory submodule and uploaded to the database;

[0048] S3, when the speech is converted into text, the recognition unit identifies whether the converted text has errors, and if the recognized text has errors, the text is cleaned up;

[0049] Specifically, when the converted text is correct, the subsequent conversion work will continue. When the converted text has errors, the erroneous text needs to be cleaned up to ensure that the text converted by the user is correct.

[0050] S4, recording the number of errors through the recording submodule, and transmitting the signal to the analysis submodule, determining the error rate of the user's speech-to-text conversion, and analyzing the quality of the speech-to-text conversion in combination with the external noise level;

[0051] S5. In order to ensure the memory of the database, delete useless data.

[0052] Specifically, since the data in the database is frequently updated, the periodic statistics submodule is used to count the time when the data is not called. If some data is not used within a period, it means that the data is useless. The clearing submodule deletes the data that is not used within a period to ensure that the database has sufficient memory.

[0053] S3 includes the following specific steps:

[0054] S31, when the recognition unit recognizes that there is an error in the text, a signal is transmitted to the extraction unit, and the extraction unit searches for the corresponding text segment in the database to correct the current erroneous text;

[0055] Specifically, since the language habits of each user are inconsistent and the user's language habits are recorded through the memory submodule, when an error occurs during the speech-to-text conversion process, the user's language habits are learned through the learning unit, and the relevant text is automatically extracted from the database according to the user's language habits, and the errors are corrected in time;

[0056] S32. In order to ensure the accuracy of correction, the cleaning submodule is started regularly to clean the text converted from the speech and remove the erroneous text;

[0057] Specifically, when the cleaning sub-module is working, it also learns the user's language habits and so on through the learning unit, screens and cleans the overall converted text according to the user's language habits, and screens out the repeated or problematic parts. At this time, they are not yet cleared, but the corresponding parts are marked for easy observation of the error-prone areas. When the user approves the problematic parts screened out, the cleaning sub-module continues to work to correct the text;

[0058] When the user has doubts about the problematic parts screened out, the manual operation sub-module is activated to facilitate manual modification by the operator to ensure the accuracy of the cleaned data;

[0059] S33. Transmit the cleaned data to the data update sub-module through the cleaning module to facilitate real-time updating of the data to the database and ensure the perfection of the database;

[0060] Specifically, based on step S32, whether the user approves or does not approve the problematic parts screened out, the operation records of the user are transmitted to the data update sub-module, enabling the database to understand the user's commonly used language (keywords) in real time, facilitating better extraction of relevant data when the user converts speech to text again later, further improving the correction efficiency, and eliminating the need for manual operation each time to achieve the function of cleaning data.

[0061] S4 includes the following specific operation steps:

[0062] S41. The analysis sub-module calculates the error rate that occurs until the user's speech-to-text conversion is completed. The formula is:

[0063] where c is the error rate that occurs until the user's speech-to-text conversion is completed, n 总 is the total number of words, and n 错 is the number of errors that occur in the speech-to-text conversion. When 0 < c < 10%, it indicates that the error rate is low level. When c ≥ 10%, it indicates that the error rate is high level;

[0064] S42. Judge the quality of the user's speech-to-text conversion;

[0065] It should be noted that the noise filtering unit identifies the magnitude of the noise while filtering the noise;

[0066] Specifically, if the error rate is low level, it indicates that the quality of the user's speech-to-text conversion is high level. If the error rate is high level and the external noise is high during the speech-to-text conversion process, the signal is transmitted to the prompt sub-module to inform the user of the reason for judging that the quality of the speech-to-text conversion is high level;

[0067] The judgment method is: inform the user through the prompt submodule that the speech-to-text error rate is high at this time, and the external noise is large during the speech-to-text conversion process, and the user repeats steps S1-S4 in a quiet environment to judge the error rate in a quiet environment. If the error rate is low in a quiet environment, it means that it is affected by the environment, and it also means that there is a problem with the quality of the noise filtering unit;

[0068] If the error rate is high in a quiet environment, it means that the quality of user speech-to-text conversion is poor, which results in a poor user experience. In this case, the signal can be transmitted to the background for processing by staff.

[0069] Through the above steps, when the work of converting speech to text is completed, the quality of the speech to text conversion can be judged to improve the user experience.

[0070] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, an electrical connection, or mutual communication; it can be a direct connection, internal communication between two elements, or an interaction relationship between two elements. For ordinary technicians in this field, the meanings of the above terms in this application can be understood according to specific circumstances.

[0071] The above is a detailed introduction to an automated data cleaning system based on big data services provided by an embodiment of the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the technical solution and its core idea of ​​the present application; ordinary technicians in this field should understand that they can still modify the technical solutions recorded in the aforementioned embodiments, or replace some of the technical features therein with equivalents; and these modifications or replacements do not make the essence of the corresponding technical solution deviate from the scope of the technical solution of the embodiments of the present application.

Claims

1. An automated data cleaning system based on big data services, comprising an automated data cleaning system and a display terminal, characterized in that: The automatic cleaning data system includes a data processing module, a database and a cleaning module; The data processing module is used to receive the user's voice and convert the voice into text; The database is used to store data; The cleaning module is used to clean up the errors in the speech-to-text conversion to ensure the accuracy of the user's speech-to-text conversion.

2. The automatic data cleaning system based on big data service according to claim 1, characterized in that: The data processing module includes a voice input submodule, a text conversion submodule, a recording submodule and a prompt submodule. The voice input submodule is used to input the user's voice, the text conversion submodule is used to convert the user's voice into text, the recording submodule is used to count the number of errors in text conversion, and the prompt submodule is used to give prompts to the user. The voice input submodule includes a noise filtering unit to filter the surrounding noisy sounds; The text conversion submodule includes a recognition unit for recognizing whether there is an error in the speech-converted text. The voice input submodule and the text conversion submodule are electrically connected to the display terminal.

3. The automatic data cleaning system based on big data service according to claim 2 is characterized by: The database includes an analysis submodule, a data update submodule and a memory submodule. The analysis submodule is electrically connected to the recording submodule and is used to analyze the error rate of speech-to-text conversion. The data update submodule is used to update the error situation of speech-to-text conversion to the database. The recording submodule is used to record the user's usage habits. The memory submodule includes a learning unit and an extracting unit, which are used to learn the user's usage habits.

4. The automatic data cleaning system based on big data service according to claim 3 is characterized by: The cleaning module includes a cleaning submodule and a manual operation submodule. The cleaning submodule is used for preliminary cleaning of data, and the manual operation submodule is used for manual operation screening.

5. The automatic data cleaning system based on big data service according to claim 4 is characterized by: The database also includes a period statistics submodule and a clearing submodule. The period statistics submodule is used to count the period of data in the database, and the clearing submodule is used to delete infrequently used data to reserve more memory for the database.

6. The automatic data cleaning system based on big data service according to claim 5, characterized in that: The automated data cleaning system includes the following specific operation steps: S1. The voice of the user is recorded through the voice input submodule, and then the voice of the user is converted into text through the text conversion submodule, and the text is displayed on the display terminal. When the voice input submodule is used, the noise filtering unit is activated to filter out external noise; S2, when the voice input submodule records the user's voice, the recording submodule records the user's language habits and transmits them to the memory submodule; S3. When the speech is converted into text, the recognition unit identifies whether the converted text has errors. If the recognized text has errors, the text is cleaned up. S4, recording the number of errors through the recording submodule, and transmitting the signal to the analysis submodule, determining the error rate of the user's speech-to-text conversion, and analyzing the quality of the speech-to-text conversion in combination with the external noise level; S5. In order to ensure the memory of the database, delete useless data.

7. The automatic data cleaning system based on big data service according to claim 6, characterized in that: The S3 includes the following specific operation steps: S31, when the recognition unit recognizes that there is an error in the text, a signal is transmitted to the extraction unit, and the extraction unit searches for the corresponding text segment in the database to correct the current erroneous text; S32. In order to ensure the accuracy of correction, the cleaning submodule is started regularly to clean the text converted from the speech and remove the erroneous text; S33, the cleaned data is transmitted to the data updating submodule through the cleaning module, so as to update the data to the database in real time and ensure the perfection of the database.

8. The automatic data cleaning system based on big data service according to claim 7, characterized in that: The S4 includes the following specific operation steps: S41, the analysis submodule calculates the error rate that occurs when the user's voice-to-text conversion is completed, and the formula is: where c is the error rate that occurs until the user's speech is converted to text successfully, and n 总 is the total number of words, and n 错 is the number of errors that occur in the conversion of language to text. When 0 < c < 10%, it indicates that the error rate is low. When c ≥ 10%, it indicates that the error rate is high; S42: Determine the quality of the user language-to-text conversion.

Citation Information

Patent Citations

  • Input system and method for conversion to characters from voice for computer

    CN109036423A

  • Vehicle interaction method and device, electronic equipment, storage medium and vehicle

    CN117235320A

  • Speech recognition method and speech recognition device

    US20200005774A1