Method, device, electronic device, computer-readable storage medium and computer program for determining recognized text to be displayed

Abstract

To provide a method, an apparatus, an electronic device and a storage medium for determining a displayed recognized text.SOLUTION: A method may include: determining a to-be-recognized data stream from a collected data stream of user recording; performing local recognition on the to-be-recognized data stream to obtain an offline recognized text; performing online recognition on the to-be-recognized data stream to obtain an online recognized text; and determining, as a displayed recognized text, one of the online and offline recognized texts better characterizing a user intention on the basis of a result of comparison between the online recognized text and the offline recognized text.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present application relates to the field of computer technology, in particular to the field of artificial intelligence technology such as speech recognition and natural language processing, in particular to a method, an apparatus, an electronic device, a computer-readable storage medium and a computer program for determining a recognized text to be displayed. [Background technology]

[0002] Car Machine is an abbreviation for car infotainment products installed in cars, and in terms of functionality, Car Machines are able to realize information communication between people and cars, and between cars and the outside world (car-to-car).

[0003] When a car machine performs speech recognition processing on the recorded data by a user, the speech recognition can be performed online using only an online speech recognition server, or offline using only a local speech recognition model. Some car machines have significant delays in processing and transmitting recorded data and updating video frames. Generally, the processing delay of recorded data can reach 500-1000ms, while the updating delay of video frames can reach 200-300ms. Summary of the Invention [Means for solving the problem]

[0004] SUMMARY OF THE DISCLOSURE Embodiments of the present disclosure provide a method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program for determining recognized text to be displayed.

[0005] In the first aspect Leave, An embodiment of the present application is a method for determining the recognized text to be displayed, based on a collected user recorded data stream. Based onThe present invention provides a method for determining a data stream to be recognized, the method including the steps of: determining a data stream to be recognized; locally recognizing the data stream to be recognized to obtain an offline recognized text; recognizing the data stream to be recognized online to obtain an online recognized text; and determining, based on a comparison result between the online recognized text and the offline recognized text, a recognized text among the online recognized text and the offline recognized text that better characterizes a user's intention, as a recognized text to be displayed.

[0006] In the second aspect Leave, An embodiment of the present application is a device for determining a recognized text to be displayed, the device being configured to recognize a collected user recorded data stream. Based on The present invention provides an apparatus including: a data stream to be recognized determination module configured to determine a data stream to be recognized; an offline recognition module configured to locally recognize the data stream to be recognized to obtain an offline recognized text; an online recognition module configured to recognize the data stream to be recognized online to obtain an online recognized text; and a displayed recognized text determination module configured to determine, based on a comparison result between the online recognized text and the offline recognized text, a recognized text that well characterizes a user's intention from among the online recognized text and the offline recognized text, as a recognized text to be displayed.

[0007] In the third aspect Leave, An embodiment of the present application provides an electronic device comprising at least one processor and a memory connected in communication with the at least one processor, wherein commands executed by the at least one processor stored in the memory, when executed by the at least one processor, cause the at least one processor to perform a method according to any of the first aspects above.

[0008] In the fourth aspect Leave,An embodiment of the present application provides a non-transitory computer readable recording medium having computer instructions recorded thereon, the computer instructions causing a computer to perform a method according to any of the first aspects above.

[0009] In the fifth aspect Leave, An embodiment of the present application is a computer program product which, when executed by a processor, causes the processor to perform a method according to any one of the first aspects described above, Computer programs provide.

[0010] The present application provides a method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program for determining a recognized text to be displayed, which first collects a user-recorded data stream. Based on After determining the data stream to be recognized, the data stream to be recognized is recognized locally to obtain offline recognized text, and then the data stream to be recognized is recognized online to obtain online recognized text, and finally, based on the comparison result between the online recognized text and the offline recognized text, the recognized text among the online recognized text and the offline recognized text that best characterizes the user's intention is determined as the recognized text to be displayed. By adopting a scheme of performing online recognition and offline recognition in parallel, the speed of offline recognition can be fully utilized, but the accuracy of online recognition can also be achieved, so that not only the problem of slow presentation of recognized text can be solved, but also the recognition result can be guaranteed.

[0011] It should be understood that the contents described in this section are not intended to identify key points or important features of the embodiments of the present disclosure, and are not used to limit the scope of the present disclosure. Other configurations of the present disclosure will be readily understood by the following description.

[0012] Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of the non-limiting examples given below with reference to the drawings, which are used for a better understanding of the present invention and are not intended to limit the present application. [Brief description of the drawings]

[0013] [Figure 1] 1 is an exemplary system architecture to which embodiments of the present disclosure can be applied. [Diagram 2] 4 is a flowchart of an embodiment of a method for determining the recognized text to be displayed according to the present disclosure. [Diagram 3] 10 is a flowchart of another embodiment of a method for determining a recognized text to be displayed according to the present disclosure. [Figure 4] 1 is a schematic diagram of an embodiment of a device for determining a recognized text to be displayed according to the present disclosure; [Diagram 5] FIG. 2 is a block diagram of an electronic device according to a method for determining a recognized text to be displayed according to an embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] The present application will be described in more detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific examples described herein are merely for the purpose of illustrating the related invention, and are not intended to limit the invention. It should also be noted that for the sake of convenience, only parts related to the invention are shown in the accompanying drawings.

[0015] It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without causing any contradiction. The present application will be described in detail below with reference to the drawings in conjunction with the embodiments.

[0016] FIG. 1 shows an exemplary system architecture 100 according to an embodiment to which the method for determining a displayed recognized text or the device for determining a displayed recognized text according to the present application can be applied.

[0017] 1, system architecture 100 may include terminal equipment 101, network 102, and server 103. Network 102 is a medium for providing a communication link between terminal equipment 101 and server 103. Network 102 may include various connection forms such as wired or wireless communication lines, fiber optic cables, and the like.

[0018] The terminal device 101 interacts with the server 103 via the network 102. The terminal device 101 can be provided with a data stream recorded by a user, an offline recognized text, etc., including but not limited to a car machine, etc.

[0019] The server 103 may be a server that provides various services, for example, the server 103 may perform processing such as analysis on data such as a user-recorded data stream and offline recognized text obtained from the terminal device 101, and generate a processing result (e.g., determining the recognized text to be displayed).

[0020] It should be noted that the server 103 may be hardware or software. If the server 103 is hardware, it may be implemented as a distributed server cluster consisting of multiple servers, or it may be implemented as a single server. If the server 103 is software, it may be implemented as multiple software programs or software modules (e.g., used to provide distributed services) or it may be implemented as a single software program or software module, but is not specifically limited thereto.

[0021] It should be noted that the method for determining the recognized text to be displayed provided in the embodiment of the present application is generally implemented by the server 103. Correspondingly, the device for determining the recognized text to be displayed is generally provided in the server 103.

[0022] It should be understood that the number of terminal devices, networks and servers shown in Figure 1 is just an example, and any number of terminal devices, networks and servers can be provided according to actual needs.

[0023] Reference is now made to Fig. 2. Fig. 2 shows a flow 200 of an embodiment of a method for determining a recognized text to be displayed according to the present application. The method includes the following steps:

[0024] In step 201, a stream of recorded data by a user is collected. Based on , the data stream to be recognized is determined.

[0025] In this embodiment, the terminal device 101 shown in FIG. 1 receives collected user recorded data streams. Based on , the data stream to be recognized is determined.

[0026] In addition, the user speaks in the microphone interception area, the user's voice is intercepted, a user's recorded data stream is generated, and then the user's recorded data stream is transmitted to the terminal device 101 via the link. The terminal device 101 includes a front-end processing module that performs front-end processing on the user's recorded data stream before feature extraction, so that the processed data can better reflect the essential characteristics of the user's voice.

[0027] Front-end processing may include, but is not limited to, endpoint detection, which distinguishes between speech and non-speech periods in a user recorded data stream to accurately determine the beginning of a speech signal. After endpoint detection, subsequent processing can be performed on the speech signal only.

[0028] The data stream to be recognized may be a speech signal after front-end processing.

[0029] In step 202, the data stream to be recognized is locally recognized to obtain offline recognized text.

[0030] In this embodiment, the terminal device 101 shown in FIG. 1 locally recognizes a data stream to be recognized, and obtains an offline recognized text.

[0031] The data stream to be recognized can be sent to a local recognition engine for local recognition. The data stream to be recognized is then recognized by a local speech model to obtain an offline recognized text. The accuracy of the offline recognition depends on the accuracy of the local speech model. The language model is trained using a large amount of text and can utilize the statistical laws of a particular language itself to improve the recognition accuracy.

[0032] After the data stream to be recognized is locally recognized to obtain offline recognized text, the offline recognized text can be transmitted to the server 103 via the network 102 .

[0033] In step 203, the data stream to be recognized is recognized online to obtain online recognized text.

[0034] In this embodiment, an executing entity (eg, the server 103 in FIG. 1) according to the method for determining the recognized text to be displayed recognizes the data stream to be recognized online to obtain the online recognized text.

[0035] In addition, the above execution subject can receive a data stream to be recognized uploaded by the terminal device 101, and then transmit the data stream to be recognized to an online recognition engine for online recognition. Next, the data stream to be recognized is recognized by an online speech model to obtain an online recognized text. In addition, the online speech model can include an acoustic model and a language model. The acoustic model usually uses a Hidden Markov Model (HMM). The acoustic model includes BLSTM and deep CNN, and fusion of different acoustic models, for example, fusion of the output layer of BLSTM and deep CNN, also helps to improve the recognition rate. The language model includes LSTM, RNN, and n-gram (n is a positive integer).

[0036] In step 204, based on the comparison result between the online-recognized text and the offline-recognized text, one of the online-recognized text and the offline-recognized text that better characterizes the user's intention is determined as the recognized text to be displayed.

[0037] In this embodiment, based on the comparison result between the online recognition text and the offline recognition text, the executing entity determines the recognition text that best characterizes the user's intention among the online recognition text and the offline recognition text as the recognition text to be displayed.

[0038] The embodiment of the present application adopts an execution mode in which offline recognition and online recognition are performed in parallel, so that at the same time, there may be two recognized texts, an online recognized text and an offline recognized text, in the execution entity. Based on this, the execution entity can arbitrate and determine which of the online recognized text and the offline recognized text is more suitable for display on a display device, so that a user can have a better experience. Specifically, an arbitration institution is set up in the execution entity, and after the offline recognized text and the online recognized text are returned to the arbitration institution, the arbitration institution can make a judgment based on a preset arbitration rule and finally select which recognized text is to be displayed on a display device.

[0039] The arbitration rules are that when the speech recognition accuracy of both recognized texts reaches an accuracy threshold, the presentation speed of the recognized text (i.e., display on a display device) takes priority; otherwise, the accuracy of the recognized text takes priority (i.e., the recognized text with the higher speech recognition accuracy is displayed first).

[0040] In addition, the arbitration rules may include determining, based on a comparison result between the online-recognized text and the offline-recognized text, which of the online-recognized text and the offline-recognized text best characterizes the user's intention as the recognized text to be displayed. For example, if the similarity between the online-recognized text and the offline-recognized text is 100%, the offline-recognized text is determined as the recognized text that characterizes the user's intention. As another example, if the similarity between the online-recognized text and the offline-recognized text is 0%, the online-recognized text is determined as the recognized text that characterizes the user's intention.

[0041] The method of determining the displayed recognized text provided in the above embodiment of the present application, when speech recognition is fused and recognized, can only use online recognition or only use offline recognition, but does not allow online recognition and offline recognition to be used simultaneously in parallel, is abandoned. By adopting the scheme of parallel online recognition and offline recognition, not only can the speed of offline recognition be fully utilized, but also the accuracy of online recognition is maintained, so that not only the problem of slow presentation of recognized text is solved, but also the recognition effect is guaranteed.

[0042] In some alternative implementations of this embodiment, step 202 above further includes a step of presenting the offline-recognized text on a display device, and step 204 above includes a step of presenting the offline-recognized text displayed on the display device as is in response to the offline-recognized text including the online-recognized text, and a step of changing the offline-recognized text displayed on the display device to the online-recognized text in response to the offline-recognized text not including the online-recognized text.

[0043] The display device may be integrated in the terminal device 101. After the executing subject determines the recognized text to be displayed, the recognized text may be output to the display device in the terminal device 101 for display. When the executing subject determines the offline recognized text as the recognized text to be displayed, the offline recognized text displayed on the display device may be displayed as is. When the executing subject determines the online recognized text as the recognized text to be displayed, the offline recognized text displayed on the display device may be changed to the online recognized text.

[0044] It should be noted that, when the offline-recognized text includes the online-recognized text, the online-recognized text may be the same as the offline-recognized text, or the online-recognized text may be a word or a paragraph in the offline-recognized text. For example, when the online-recognized text is "play music", the offline-recognized text is "want to play music on my mobile phone". It should be noted that, when the offline-recognized text includes the online-recognized text, characters or words in the online-recognized text may be scattered and distributed in the offline-recognized text. For example, when the online-recognized text is "play music", the offline-recognized text is "want to play dynamic music on my mobile phone".

[0045] Generally, the speech recognition accuracy of the online recognized text is higher than the accuracy threshold. If the offline recognized text includes the online recognized text, the accuracy of the offline recognized text is also higher than the accuracy threshold. Therefore, based on the arbitration rule that "when the speech recognition accuracy of both the two recognized texts reaches the accuracy threshold, the speed of presenting the recognized text (i.e., displaying on the display device) takes precedence," the recognized text that is presented faster on the display device can be selected as the recognized text to be displayed. After performing the above step 202, the offline recognized text is presented on the display device, so the offline recognized text displayed on the display device can be presented as it is.

[0046] In addition, when the offline-recognized text includes the online-recognized text, it is possible to select that the online-recognized text is a word or a paragraph in the offline-recognized text, and the length of the online-recognized text may be less than or equal to the length of the offline-recognized text.

[0047] In some alternative implementations of this embodiment, step 204 above includes a step of determining the online-recognized text as the recognized text to be displayed in response to a similarity between the offline-recognized text and the online-recognized text being less than a predetermined threshold, and a step of determining the offline-recognized text as the recognized text to be displayed in response to a similarity between the offline-recognized text and the online-recognized text being equal to or greater than a predetermined threshold.

[0048] Generally, the speech recognition accuracy of the online recognition text is higher than the accuracy threshold. By determining the similarity between the offline recognition text and the online recognition text, it can be determined whether the speech recognition accuracy of the offline recognition text is higher than the accuracy threshold. It should be noted that the similarity threshold between the offline recognition text and the online recognition text can be set in advance, and the similarity threshold is for characterizing whether the speech recognition accuracy of the offline recognition text is higher than the accuracy threshold. For example, when the accuracy threshold is 90%, it is tacitly accepted that the speech recognition accuracy of the online recognition text is 100%, and the similarity threshold between the offline recognition text and the online recognition text can be set to 90%.

[0049] In addition, if the similarity between the offline-recognized text and the online-recognized text is less than a predetermined threshold, the accuracy of the offline-recognized text does not reach the accuracy threshold. According to the above arbitration rules, the accuracy of the recognized text is prioritized, that is, the recognized text with higher speech recognition accuracy--online-recognized text is presented preferentially. If the similarity between the offline-recognized text and the online-recognized text is equal to or greater than a predetermined threshold, the accuracy of the offline-recognized text reaches the accuracy threshold. According to the above arbitration rules, the speed of the recognized text is prioritized, that is, the recognized text with faster presentation speed--offline-recognized text is presented preferentially.

[0050] 3, a flow chart of an embodiment of a method for determining the recognized text to be displayed is shown. The method includes the following steps.

[0051] In step 301, a collected user recorded data stream is Based on , the data stream to be recognized is determined.

[0052] In step 302, the data stream to be recognized is locally recognized to obtain offline recognized text, and the offline recognized text is presented on a display device.

[0053] In step 303, the data stream to be recognized is recognized online to obtain online recognized text.

[0054] In step 304, based on the comparison result between the online-recognized text and the offline-recognized text, one of the online-recognized text and the offline-recognized text that better characterizes the user's intention is determined as the recognized text to be displayed.

[0055] In step 305, in response to no online-recognized text being returned, the offline-recognized text displayed on the display device is retained.

[0056] In this embodiment, the terminal device 101 shown in FIG. 1 can respond by simply displaying the offline-recognized text displayed on the display device when the online-recognized text is not returned.

[0057] In addition, the online recognized text may not be returned due to abnormal reasons such as poor network environment, network disconnection, failure of the online speech recognition server, etc. If the online recognized text is not returned due to an abnormal reason, an abnormality handling mechanism is provided to determine the displayed offline recognized text as the recognized text to be displayed.

[0058] In some alternative implementations of this embodiment, step 201 above includes pre-processing the collected audio recording data stream to obtain a data stream to be recognized.

[0059] The pre-processing includes noise reduction, echo cancellation, and reverberation cancellation. The noise reduction flow is to stabilize the spectral characteristics of the background noise, and the amplitude is very stable in one or more frequency spectra. For example, if a small part of the background is background noise, the noise spectrum is obtained by grouping the background noise first and then Fourier transforming it to obtain the average value of these groups. The process of noise reduction is to inversely compensate the noisy voice and then obtain the voice with reduced noise.

[0060] Incidentally, echo cancellation includes double talk detection and delay estimation. The echo cancellation module of the intelligent terminal updates the filters w1 and w2 in various ways according to the current mode (near talk mode, far talk mode, double talk mode) to eliminate the far-end interference. Then, the post-filtering algorithm eliminates the interference of residual noise. Generally, there are two types of echo: acoustic echo and mixed echo. Back echo occurs in voice communication due to voice compression technology and batch processing delay, so back echo cancellation is usually required to improve the quality of the communication. The back echo canceller monitors and measures the speech from the far end on the receiving path, calculates an estimate of the back echo, and then subtracts this estimate from the transmitting path to eliminate the back echo, and only the near-end speech is transmitted to the far end.

[0061] In acoustics, the reflected wave with a delay time of about 50 ms or more is called echo, and the remaining reflected wave effect is called reverberation, which can be divided into early reverberation and late reverberation. In the task of speech dereverberation, more attention is paid to the suppression of late reverberation, since the early reverberation component helps to improve the intelligibility of speech. There are four main categories of reverberation cancellation methods based on beamforming, inverse filtering, speech enhancement, and deep learning.

[0062] By pre-processing the collected recording data stream, the accuracy of speech recognition can be improved.

[0063] In some alternative implementations of this embodiment, the length of the offline-recognized text is below a preset threshold, for example, to ensure the presentation speed of the offline-recognized text, the return characters of the offline-recognized text are limited to a maximum of 12 characters.

[0064] Further referring to Fig. 4, as an implementation of the method shown in the above figures, the present application provides an embodiment for determining the recognized text to be displayed, the device embodiment corresponds to the method embodiment shown in Fig. 2, and the device can be applied to various electronic devices.

[0065] 4, the device 400 for determining a recognized text to be displayed according to the present embodiment includes a module 401 for determining a data stream to be recognized, a module 402 for determining an offline recognition, a module 403 for determining an online recognition, and a module 404 for determining a recognized text to be displayed. Based on, a data stream to be recognized, the offline recognition module 402 is configured to locally recognize the data stream to be recognized to obtain an offline recognized text, the online recognition module 403 is configured to online recognize the data stream to be recognized to obtain an online recognized text, and the displayed recognized text determination module 404 is configured to determine, based on a comparison result between the online recognized text and the offline recognized text, a recognized text that better characterizes the user's intention from among the online recognized text and the offline recognized text, as a displayed recognized text.

[0066] In this embodiment, in the device 400 for determining the displayed recognized text, the specific processing of the data stream determination module 401 to be recognized, the offline recognition module 402, the online recognition module 403, and the displayed recognized text determination module 404 and the technical effects thereof can be referred to the relevant descriptions of steps 201 to 205 in the embodiment corresponding to FIG. 2, so the description will be omitted here.

[0067] In some alternative implementations of this embodiment, the offline recognition module further includes a display module configured to present the offline recognition text on a display device, and the displayed recognized text determination module is further configured to present the offline recognition text displayed on a display device in response to the offline recognition text including the online recognition text.

[0068] In some alternative implementations of this embodiment, the displayed recognition text determination module is further configured to change the offline recognition text displayed on the display device to the online recognition text in response to the offline recognition text not including the online recognition text.

[0069] In some alternative implementations of this embodiment, the displayed recognized text determination module is further configured to determine the online recognized text as the displayed recognized text in response to a similarity between the offline recognized text and the online recognized text being less than a predetermined threshold.

[0070] In some alternative implementations of this embodiment, the displayed recognized text determination module is further configured to determine the offline recognized text as the displayed recognized text in response to a similarity between the offline recognized text and the online recognized text being equal to or greater than a predetermined threshold. In some alternative implementations of this embodiment, a length of the online recognized text is less than or equal to a length of the offline recognized text.

[0071] In some alternative implementations of this embodiment, the device further includes an anomaly processing module configured to, in response to the online recognized text not being returned, present the offline recognized text displayed on the display device as is.

[0072] In some alternative implementations of this embodiment, the target data stream determination module is further configured to pre-process the collected audio recording data stream to obtain the target data stream.

[0073] In some alternative implementations of this embodiment, the pre-processing includes any one of noise reduction, echo cancellation, and reverberation cancellation.

[0074] In some alternative implementations of this embodiment, the length of the offline recognized text is less than or equal to a preset threshold.

[0075] As shown in FIG. 5, a block diagram of an electronic device according to the method of determining the recognized text to be displayed according to an embodiment of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, mobile phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application as described and / or claimed herein.

[0076] As shown in FIG. 5, the electronic device includes one or more processors 501, memory 502, and interfaces including high-speed and low-speed interfaces for connecting each component. Each component is connected to each other by different buses and may be mounted on a common main board or may be mounted in other ways as needed. The processor can process commands executed in the electronic device, including commands stored in the memory or commands for displaying graphical information of a GUI on an external input / output device (e.g., a display device coupled to the interface) in the memory. In other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memories as needed. Similarly, multiple electronic devices can be connected to provide each device with some required operations (e.g., a server array, a group of blade servers, or a multi-processor system). FIG. 5 shows the processor 501 as an example.

[0077] The memory 502 is a non-transitory computer-readable storage medium provided by the present application, in which the instructions stored in the memory and executable by at least one processor cause the at least one processor to perform the method for determining the recognized text to be displayed provided by the present application. The non-transitory computer-readable storage medium of the present application stores computer commands, which cause a computer to execute the method for determining the recognized text to be displayed provided by the present application.

[0078] The memory 502 is a non-transitory computer-readable recording medium used to store non-transitory software programs, non-transitory computer-executable programs and modules, such as program commands / modules corresponding to the method for determining the displayed recognized text according to the embodiment of the present application (e.g., the recognition target data stream determining module 401, the offline recognition module 402, the online recognition module 403, and the displayed recognized text determining module 404 shown in FIG. 4). The processor 501 executes the non-transitory software programs, commands and modules stored in the memory 502 to perform various functional applications and data processing in the server, thereby realizing the method for determining the displayed recognized text according to the above method embodiment.

[0079] The memory 502 may include a program storage area and a data storage area, where the program storage area may store an operating system, an application program required by at least one function, and the data storage area may store data required by the electronic device to determine the recognized text to be displayed, etc. The memory 502 may also include high-speed random access memory and may also include non-transitory memory, such as at least one magnetic disk storage device, flash storage device, or other non-transitory solid-state storage device. In some embodiments, the memory 502 may select memories configured remotely to the processor 501, and these remote memories may be connected to the electronic device to determine the recognized text to be displayed via a network, examples of which include, but are not limited to, the Internet, a corporate intranet, a local area network, a mobile communication network, and combinations thereof.

[0080] The electronic device according to the method for determining the recognized text to be displayed may further include an input device 503 and an output device 504. The processor 501, the memory 502, the input device 503, and the output device 504 may be connected by a bus or in another manner. In FIG. 5, a bus connection is shown as an example.

[0081] The input device 503 can receive input numeric and textual information and generate key signal inputs associated with user settings and function control of the electronic device by determining how the recognized text is displayed, and may be, for example, a touch screen, a keypad, a mouse, a track pad, a touchpad, an indicator stick, one or more mouse buttons, a trackball, a joystick, etc. The output device 504 can include a display device, an auxiliary lighting device (e.g., LEDs), a tactile feedback device (e.g., vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, a plasma display, etc. In some embodiments, the display device may be a touch screen.

[0082] Various embodiments of the systems and techniques described herein may be realized in digital electronic circuitry systems, integrated circuit systems, dedicated ASICs (Application Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include being implemented in one or more computer programs that may be executed and / or interpreted by a programmable system that includes at least one programmable processor, which may be a dedicated or general purpose programmable processor, and may receive data and commands from a storage system, at least one input device, and at least one output device, and may transmit data and commands to the storage system, the at least one input device, and the at least one output device.

[0083] These computer programs (also referred to as programs, software, software applications, or codes) include machine language commands for a programmable processor, and execute these computer programs using high level process and / or object oriented programming languages, and / or assembler / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, machine, and / or apparatus (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) for providing machine commands and / or data to a programmable processor, including machine-readable medium for receiving machine commands as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine commands and / or data to a programmable processor.

[0084] A computer may implement the systems and techniques described herein to provide interaction with a user. The computer may include a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user, and a keyboard and pointing device (e.g., a mouse or trackball) through which a user provides input to the computer. Other types of devices may be used to provide interaction with a user, for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback) and may receive any form of input from the user (including acoustic, speech, or tactile input).

[0085] The systems and techniques described herein may be implemented in a computer system including a back-end component (e.g., a data server), or an intermediate component (e.g., an application server), or a front-end component (e.g., a user computer having a graphical user interface or a network browser through which a user interacts with an embodiment of the systems and techniques described herein), or any combination of back-end, intermediate, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0086] A computer system may include clients and servers. Clients and servers are typically remote from each other and typically interact through a communication network. The relationship of client and server is created by computer programs running on the corresponding computers having a client-server relationship to each other. According to the technical solutions of the embodiments of the present application, the law of nature, First, content information of the target video frame is obtained, which is beneficial for understanding the target video frame. Then, text description information is constructed based on the content information, thereby realizing information-based video description. Then, the text description information is imported into a pre-trained text conversion model to obtain commentary text information corresponding to the text description information, thereby improving the suitability of the video commentary. Finally, the commentary text information is converted into audio information. The present application realizes audio commentary for video, thereby improving the efficiency of video commentary.

[0087] According to the technical solution of the present application, first, the collected user recorded data stream Based onFirst, a data stream to be recognized is determined, the data stream to be recognized is locally recognized to obtain offline recognized text, and then the data stream to be recognized is recognized online to obtain online recognized text. Finally, based on a comparison result between the online recognized text and the offline recognized text, one of the online recognized text and the offline recognized text that best characterizes the user's intention is determined as the recognized text to be displayed. By adopting a scheme of performing online and offline recognition in parallel, the speed of offline recognition can be fully utilized, and the accuracy of online recognition can be maintained, so that not only the problem of slow presentation of recognized text can be solved, but also the recognition result can be guaranteed.

[0088] Artificial intelligence is a discipline that studies computers to simulate certain human thought processes and intelligent behaviors (learning, reasoning, thinking, planning, etc.), and includes both hardware and software technologies. In general, artificial intelligence hardware technologies include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing, etc. Artificial intelligence software technologies mainly include computer vision technology, voice recognition technology, natural language processing technology, machine learning / deep learning, big data processing technology, and knowledge encyclopedia technology.

[0089] It should be understood that various forms of flows, reordering, adding or deleting steps shown above can be utilized. For example, each step described in this application can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution disclosed in this application can be achieved, which is not limited to this specification.

[0090] The above specific embodiments do not limit the scope of the present application. It should be understood that those skilled in the art can make various modifications, combinations, subcombinations and substitutions according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application shall be included in the scope of the present application.

Claims

1. 1. A method for determining recognized text to be displayed, comprising: determining a data stream to be recognized based on the collected user recorded data streams; locally recognizing the data stream to be recognized to obtain offline recognized text; online recognizing the data stream to be recognized to obtain online recognized text; determining, based on a comparison result between the online recognition text and the offline recognition text, a recognition text that better characterizes a user's intention among the online recognition text and the offline recognition text as a recognition text to be displayed; Including, The step of locally recognizing the data stream to be recognized and obtaining offline recognized text includes: presenting the offline recognized text on a display device; The step of determining, based on the result of the comparison between the online recognition text and the offline recognition text, a recognition text that better characterizes the user's intention among the online recognition text and the offline recognition text as a recognition text to be displayed, includes: responsive to the offline-recognized text containing the online-recognized text, presenting the offline-recognized text displayed on a display device; The method, wherein the offline-recognized text includes the online-recognized text means that the online-recognized text is a word or a paragraph in the offline-recognized text, or characters or words in the online-recognized text are sparsely distributed in the offline-recognized text, and the length of the online-recognized text is smaller than the length of the offline-recognized text.

2. The step of determining, based on the result of the comparison between the online recognition text and the offline recognition text, a recognition text that better characterizes the user's intention among the online recognition text and the offline recognition text as a recognition text to be displayed, includes: in response to the offline-recognized text not including the online-recognized text, changing the offline-recognized text displayed on a display device to the online-recognized text. The method of claim 1.

3. The step of determining, based on the result of the comparison between the online recognition text and the offline recognition text, a recognition text that better characterizes the user's intention among the online recognition text and the offline recognition text as a recognition text to be displayed, includes: in response to a similarity between the offline-recognized text and the online-recognized text being less than a predetermined threshold, determining the online-recognized text as the recognized text to be displayed. The method of claim 1.

4. The step of determining, based on the result of the comparison between the online recognition text and the offline recognition text, a recognition text that better characterizes the user's intention among the online recognition text and the offline recognition text as a recognition text to be displayed, includes: determining, in response to a similarity between the offline-recognized text and the online-recognized text being equal to or greater than a predetermined threshold, the offline-recognized text as the recognized text to be displayed; The method according to claim 3.

5. in response to a failure to receive the online-recognized text, presenting the offline-recognized text displayed on the display device; The method of claim 1.

6. The step of determining a data stream to be recognized based on the collected user-recorded data streams includes: pre-processing the collected audio data stream to obtain a data stream to be recognized; The method of claim 1.

7. The pre-processing includes any one of noise reduction, echo cancellation, and reverberation cancellation. The method according to claim 6.

8. the length of the offline recognized text is less than or equal to a preset threshold; The method according to any one of claims 1 to 7.

9. 1. An apparatus for determining a recognized text to be displayed, comprising: a target data stream determination module configured to determine a target data stream based on the collected user recorded data streams; an offline recognition module configured to locally recognize the data stream to be recognized to obtain an offline recognized text; an online recognition module configured to recognize the data stream to be recognized online to obtain an online recognized text; a displayed recognized text determination module configured to determine, based on a comparison result between the online recognized text and the offline recognized text, a recognized text that better characterizes a user's intention among the online recognized text and the offline recognized text as a displayed recognized text; The offline recognition module includes: a display module configured to present the offline recognized text on a display device; The displayed recognized text determination module further comprises: configured to present the offline-recognized text displayed on a display device in response to the offline-recognized text including the online-recognized text; The offline-recognized text including the online-recognized text means that the online-recognized text is a word or a paragraph in the offline-recognized text, or characters or words in the online-recognized text are sparsely distributed in the offline-recognized text, and the length of the online-recognized text is smaller than the length of the offline-recognized text.

10. The displayed recognized text determination module further comprises: and configured, in response to the offline-recognition text not including the online-recognition text, to change the offline-recognition text displayed on a display device to the online-recognition text.

10. The apparatus of claim 9.

11. The displayed recognized text determination module further comprises: configured to determine, in response to a similarity between the offline-recognized text and the online-recognized text being less than a predetermined threshold, the online-recognized text as the recognized text to be displayed.

10. The apparatus of claim 9.

12. The displayed recognized text determination module further comprises: configured to determine, in response to a similarity between the offline-recognized text and the online-recognized text being equal to or greater than a predetermined threshold, the offline-recognized text as the recognized text to be displayed.

12. The apparatus of claim 11.

13. and an anomaly processing module configured to, in response to the online-recognized text not being returned, present the offline-recognized text displayed on the display device.

10. The apparatus of claim 9.

14. The target data stream determination module further comprises: A method for pre-processing the collected audio data stream to obtain a data stream to be recognized.

10. The apparatus of claim 9.

15. The pre-processing includes any one of noise reduction, echo cancellation, and reverberation cancellation.

15. The apparatus of claim 14.

16. the length of the offline recognized text is less than or equal to a preset threshold; Apparatus according to any one of claims 9 to 15.

17. An electronic device, At least one processor; a memory connected in communication with the at least one processor; The instructions executed by the at least one processor stored in the memory, when executed by the at least one processor, cause the at least one processor to perform the method of any one of claims 1 to 8.

18. A non-transitory computer-readable recording medium having a computer program recorded thereon, The computer program is a non-transitory computer-readable recording medium that causes a computer to execute the method according to any one of claims 1 to 8.

19. A computer program comprising: A computer program which, when executed by a processor, causes the computer program to carry out the method according to any one of claims 1 to 8.

Images