Speech ability data acquisition method and audio and video synchronous acquisition system

Abstract

The invention provides a speech ability data acquisition method and an audio and video synchronous acquisition system, the audio and video synchronous acquisition system comprises a handheld equipment main body, an audio acquisition module and a video acquisition module, the audio acquisition module and the video acquisition module are arranged on the handheld equipment main body, and an equipment processor is integrated in the equipment main body, the method comprises the following steps: in response to a synchronous acquisition starting instruction, starting an audio acquisition module and a video acquisition module at the same time; recording audio data sent by a test object through an audio acquisition module and recording a face video of the test object through a video acquisition module; performing synchronous processing on the audio data and the face video; and storing the audio data and the face video after synchronization processing. According to the invention, an audio acquisition function and a video acquisition function are integrated into the same system and data synchronization processing is carried out, so that the related operation process of speech ability detection is simplified, and the subsequent evaluation precision is improved through multi-modal data acquisition.

Description

Technical Field

[0001] This application relates to the field of children's speech ability testing technology, and in particular to a speech ability data acquisition method and an audio-visual synchronous acquisition system. Background Technology

[0002] Speech development is a crucial stage in a child's growth, and early detection and intervention of speech disorders are essential for a child's healthy development. Traditional methods of measuring speech ability often rely on recording equipment to assess a child's speech by analyzing audio indicators such as voice, speech rate, and fluency. However, these methods have limitations. They often ignore the physical movements involved in pronunciation, such as lip shape, degree of opening and closing, and movement patterns. Pronunciation is a complex process involving hearing and movement, and lip shape is a key visual component. For example, for some children with pronunciation difficulties, the problem may stem from incorrect lip muscle movements, which is difficult to diagnose accurately through audio analysis alone.

[0003] Existing speech ability assessment devices generally only acquire audio and cannot simultaneously capture lip movements, resulting in inaccurate subsequent speech ability analysis results. In cases where audio and video capture coexist in existing technologies, the devices are independent of each other, resulting in significant deficiencies in data synchronization. Furthermore, the devices are usually arranged separately, requiring test subjects to manage different devices simultaneously, increasing the complexity of the testing process and adding to the interference and discomfort experienced by test personnel due to managing multiple devices. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide at least one speech ability data acquisition method and audio-video synchronous acquisition system, which integrates audio acquisition function and video acquisition function into the same system and performs data synchronous processing, simplifies the speech ability detection related operation process, and improves the accuracy of subsequent assessment by multimodal data acquisition.

[0005] This application mainly includes the following aspects: In a first aspect, embodiments of this application provide a speech ability data acquisition method, applied to a device processor within an audio-video synchronous acquisition system. The audio-video synchronous acquisition system includes a handheld device body, an audio acquisition module, and a video acquisition module. The audio acquisition module and the video acquisition module are disposed on the handheld device body, and the device processor is integrated within the handheld device body. The method includes: simultaneously activating the audio acquisition module and the video acquisition module in response to a synchronous acquisition start command; recording audio data emitted by a test subject through the audio acquisition module and recording facial video of the test subject through the video acquisition module; synchronously processing the audio data and facial video; and saving the synchronously processed audio data and facial video.

[0006] In one possible implementation, the audio data and facial video are synchronized by: performing image processing on the facial video to generate a video of the test subject's mouth movements; and adding a unified timestamp to both the mouth movement video and the facial video.

[0007] In one possible implementation, the facial video includes multiple facial video frames, wherein a mouth movement video is generated by: for each facial video frame, performing: extracting a mouth feature image of the test subject from the facial video frame, using the mouth feature image as the mouth feature video frame corresponding to the facial video frame, and replacing the facial video frame with the mouth feature video frame; and generating a mouth movement video of the test subject from each mouth feature video frame.

[0008] In one possible implementation, the mouth movement video includes multiple mouth feature video frames, and the audio data includes multiple audio frames. The step of adding a unified timestamp to the mouth movement video and the facial movement video includes adding the same timestamp to the mouth feature video frames and audio frames at the same acquisition time.

[0009] In one possible implementation, the audio and video synchronization acquisition system further includes a data processing terminal, which is connected to the device processor via a preset communication method. The method further includes: receiving a data reading instruction transmitted by the data processing terminal; and in response to the data reading instruction, sending the synchronized audio data and facial video to the data processing terminal.

[0010] In one possible implementation, the audio acquisition module and the video acquisition module are arranged adjacent to each other.

[0011] In one possible implementation, the optical axis of the video acquisition module is parallel to the optimal pickup axis of the audio acquisition module.

[0012] Secondly, embodiments of this application also provide a speech ability data acquisition device. The device is applied to a device processor within an audio-video synchronous acquisition system. The audio-video synchronous acquisition system includes a handheld device body, an audio acquisition module, and a video acquisition module. The audio acquisition module and video acquisition module are disposed on the handheld device body, and the device processor is integrated within the handheld device body. The device includes: an instruction response module, used to simultaneously activate the audio acquisition module and the video acquisition module in response to a synchronous acquisition start instruction; an acquisition module, used to record audio data emitted by the test subject through the audio acquisition module and to record facial video of the test subject through the video acquisition module; a synchronization module, used to synchronously process the audio data and facial video; and a storage module, used to store the synchronously processed audio data and facial video.

[0013] Thirdly, this application also provides an audio and video synchronous acquisition system, which includes a handheld device body, an audio acquisition module and a video acquisition module. The audio acquisition module and the video acquisition module are disposed on the handheld device body, and the device processor is integrated in the device body. The device processor runs the speech ability data acquisition method provided in any of the above embodiments.

[0014] Fourthly, this application also provides a data processing terminal, which sends a data reading instruction to the device processor in the audio and video synchronization acquisition system provided in any of the above embodiments; receives and saves the synchronized audio data and facial video fed back by the device processor. This application provides a speech ability data acquisition method and a synchronized audio-video acquisition system. The synchronized audio-video acquisition system includes a handheld device body, an audio acquisition module, and a video acquisition module. The audio acquisition module and the video acquisition module are disposed on the handheld device body, and the device processor is integrated within the device body. The method includes: simultaneously activating the audio acquisition module and the video acquisition module in response to a synchronized acquisition start command; recording audio data emitted by the test subject through the audio acquisition module and recording facial video of the test subject through the video acquisition module; synchronously processing the audio data and facial video; and saving the synchronized audio data and facial video. This application simplifies the speech ability testing process by integrating audio acquisition and video acquisition functions into the same system and performing synchronized data processing, and improves the accuracy of subsequent assessments through multimodal data acquisition.

[0015] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This application illustrates one of the audio and video synchronization acquisition systems provided in its embodiments; Figure 2 A flowchart of a speech ability data acquisition method provided in an embodiment of this application is shown; Figure 3 This illustration shows a second example of an audio-visual synchronization acquisition system provided in this application. Figure 4A functional block diagram of a speech ability data acquisition device provided in an embodiment of this application is shown. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0019] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0020] Speech development is a crucial stage in a child's growth, and early detection and intervention of speech disorders are essential for a child's healthy development. Traditional methods of measuring speech ability often rely on recording equipment to assess a child's speech by analyzing audio indicators such as voice, speech rate, and fluency. However, these methods have limitations. They often ignore the physical movements involved in pronunciation, such as lip shape, degree of opening and closing, and movement patterns. Pronunciation is a complex process involving hearing and movement, and lip shape is a key visual component. For example, for some children with pronunciation difficulties, the problem may stem from incorrect lip muscle movements, which is difficult to diagnose accurately through audio analysis alone.

[0021] Therefore, it is evident that the existing technology has at least the following drawbacks: 1. The speech ability assessment equipment provided can generally only obtain audio and cannot simultaneously capture lip movements, resulting in inaccurate final analysis results.

[0022] 2. Furthermore, existing technologies generally involve fixing the audio acquisition device within a certain range of the test subject. When the test subject is a child, their behavior is uncontrollable, and they may move around randomly, resulting in poor sound reception.

[0023] 3. In the case of existing technologies where audio and video capture devices coexist, the devices are independent of each other, resulting in obvious defects in data synchronization. Furthermore, the devices are usually arranged separately, requiring the test subject to take into account different devices simultaneously, which increases the complexity of the testing process and increases the interference and discomfort caused to the test personnel due to taking care of multiple devices.

[0024] Based on this, embodiments of this application provide a method for acquiring speech ability data and a synchronized audio-video acquisition system. By integrating audio and video acquisition functions into the same system and performing synchronized data processing, the operation process related to speech ability testing is simplified, and multimodal data acquisition improves the accuracy of subsequent assessments. Specifically, as follows: Please see Figure 1 , Figure 1 This application illustrates one type of audio and video synchronization acquisition system provided in its embodiments. For example... Figure 1 As shown, the audio and video synchronous acquisition system provided in this application embodiment includes a handheld device body 1 (exemplarily, a handheld microphone body), an audio acquisition module 2, and a video acquisition module 3. The audio acquisition module 2 and the video acquisition module 3 are disposed on the handheld device body 1. The device processor (not shown in the figure, but exemplarily, such as a microcontroller) is integrated in the handheld device body 1. The handheld device body 1, the audio acquisition module 2, the video acquisition module 3, and the device processor form an integrated audio and video acquisition device.

[0025] In a preferred embodiment, such as Figure 1 As shown, the main body 1 of the handheld device includes an integrated acquisition head 11 and a handheld handle 12. The audio acquisition module 2 and the video acquisition module 3 are mounted on the acquisition head 11 and are arranged adjacent to each other.

[0026] The audio acquisition module 2 is used to acquire audio data emitted by the test subject. The audio acquisition module 2 includes at least one high-fidelity microphone. The video acquisition module 3 is used to capture facial images (specifically, mouth feature images) of the test subject when it speaks. The video acquisition module 3 includes a high-definition camera.

[0027] Preferably, the optical axis of the video acquisition module 3 is parallel to the optimal pickup axis of the audio acquisition module 2. This is to ensure that the alignment direction of the video acquisition module 3 is consistent with the pickup direction of the audio acquisition module 2, so that when the test subject speaks to the integrated audio and video acquisition device, his / her voice and facial image can be captured simultaneously for optimal capture.

[0028] In one specific embodiment, please refer to Figure 2 , Figure 2A flowchart illustrating a speech ability data acquisition method provided in an embodiment of this application is shown. Figure 2 As shown, the speech ability data acquisition method provided in this application is applied to the device processor within an integrated audio and video acquisition device, wherein the method includes: S100: In response to the synchronous acquisition start command, the audio acquisition module and the video acquisition module are started simultaneously.

[0029] S101. Record the audio data emitted by the test subject through the audio acquisition module and record the facial video of the test subject through the video acquisition module.

[0030] S102. Synchronize the audio data and facial video.

[0031] S103. Save the synchronized audio data and facial video.

[0032] In a specific implementation, in step S100, the handheld device body 1 also integrates a start button connected to the device processor. In response to the press of the start button, a corresponding synchronous acquisition start command is generated and sent to the device processor.

[0033] In step S101, the audio acquisition module 2 and the video acquisition module 3 are started synchronously. After starting, the audio data emitted by the test subject is recorded by the audio acquisition module 2, and the facial video of the test subject is recorded by the video acquisition module 3. At each acquisition moment, the audio data acquired by the audio acquisition module 2 corresponds to the audio data acquired by the video acquisition module 3.

[0034] In a preferred embodiment, step S102 includes: S1021. Perform image processing on the facial video to generate a video of the test subject's mouth movements.

[0035] S1022. Add a unified timestamp to the mouth movement video and the face video.

[0036] In steps S1021 to S1022, the shape, opening and closing degree, and movement pattern of the lips are considered as influencing factors affecting the occurrence of speech. The mouth movement characteristics of the test subject during speech need to be effectively collected for further analysis. After the video acquisition module 3 of this application is started, it generally collects the facial video (i.e., full-face features) of the test subject. Therefore, based on the facial video, this application performs local feature extraction on the facial video to generate the mouth movement video corresponding to the test subject, and then unifies the timestamps of the converted mouth movement video and the facial video.

[0037] In a preferred embodiment, the facial video includes multiple facial video frames, wherein step S1021 includes: For each facial video frame, the following steps are performed: extract the mouth feature image of the test subject from the facial video frame, use the mouth feature image as the mouth feature video frame corresponding to the facial video frame, replace the facial video frame with the mouth feature video frame, and generate a mouth movement video of the test subject from each mouth feature video frame.

[0038] In one example, this application extracts mouth features from the acquired facial video frame by frame, and the resulting mouth feature video frame is still associated with the acquisition time corresponding to its respective facial video frame. That is, the facial video frame to which the mouth feature video frame belongs is replaced, which facilitates subsequent analysis based directly on the mouth features of the tester.

[0039] In another preferred embodiment, the audio data includes multiple audio frames, wherein the step of adding a unified timestamp to the mouth movement video and the facial motion video includes: adding the same timestamp to the mouth feature video frame and audio frame at the same acquisition time.

[0040] In this application, due to the hardware positional relationship between the audio acquisition module 2 and the video acquisition module 3, the audio and video synchronous acquisition system provided by this application can basically guarantee the synchronization of mouth feature video frames and audio frames at the same acquisition time. Therefore, this application can add the same timestamp to the mouth feature video frames and audio frames at the same acquisition time based on the same clock source. The timestamp added here refers to PTS (Presentation Time Stamp), thus completing the synchronous processing of mouth feature video frames and audio frames.

[0041] Please see Figure 3 , Figure 3 This illustration shows a second example of an audio / video synchronization acquisition system provided in this application. For instance... Figure 3 As shown, the audio and video synchronous acquisition system also includes a data processing terminal PC. The data processing terminal PC is connected to the device processor 4 via a preset communication method (e.g., wireless Bluetooth, data cable, etc.). The method provided in this application further includes: It receives a data read command from the data processing terminal PC, and in response to the data read command, sends the synchronized audio data and mouth movement video to the data processing terminal PC.

[0042] In this application, the processing power of the device processor 4 is limited. Therefore, this application establishes a data interaction channel between the data processing terminal PC and the device processor 4, so as to periodically read the corresponding synchronized audio data and mouth movement video from the device processor 4 through the preset communication method indicated by the data interaction channel.

[0043] In another specific embodiment, after the data processing terminal PC obtains the synchronized audio data and mouth movement video corresponding to the test subject, it can run the preset speech ability assessment model built in it. The synchronized audio data and mouth movement video are used as the input of the preset speech ability assessment model. The preset speech ability assessment model performs joint analysis on the input data from audio features (e.g., factors, pitch) and mouth features (e.g., lip opening and closing, lip movement pattern, etc.) and outputs a speech ability assessment report corresponding to the test subject. The speech ability assessment report indicates the speech ability assessment result of the test subject.

[0044] The method provided in this application has the following advantages: Based on the same application concept, this application also provides a speech ability data acquisition device corresponding to the speech ability data acquisition method provided in the above embodiments. Since the principle of the device in this application is similar to the speech ability data acquisition method in the above embodiments of this application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0045] Please see Figure 4 , Figure 4 This diagram illustrates the functional block diagram of a speech ability data acquisition device provided in an embodiment of this application. Figure 4 As shown, the device includes: The instruction response module 500 is used to respond to the synchronous acquisition start instruction and simultaneously start the audio acquisition module and the video acquisition module; The acquisition module 510 is used to record audio data emitted by the test subject through the audio acquisition module and to record facial video of the test subject through the video acquisition module. The synchronization module 520 is used to synchronize audio data and facial video. The storage module 530 is used to save the synchronized audio data and facial video.

[0046] Based on the same concept, this application also provides a data processing terminal. The data processing terminal sends a data reading instruction to the device processor 4 in the audio and video synchronization acquisition system provided in any of the above embodiments, receives the synchronized audio data and mouth movement video fed back by the device processor 4, and saves them. The advantages of this application are: 1. The audio and video synchronous acquisition system provided in this application simplifies the operation process by integrating the camera and microphone together. The test subject only needs to speak to a single integrated device, reducing the interference and discomfort caused by multiple devices.

[0047] 2. The audio and video synchronous acquisition system provided in this application synchronously associates the audio data and mouth movement video of the test subject, which helps the subsequent data processing terminal to more accurately determine the root cause of the test subject's speech disorder, such as whether it is a breathing problem or a lip muscle coordination problem.

[0048] 3. The audio and video synchronous acquisition system provided in this application increases the dimensions of assessment data for subsequent speech ability assessment, making the final assessment results more comprehensive and in-depth.

[0049] 4. The collected lip-sync videos can also serve as visual feedback, helping speech therapists and test subjects to intuitively understand and correct incorrect pronunciation movements, providing a basis for rehabilitation training.

[0050] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0051] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0052] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0053] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A speech ability data collection method characterized by comprising: The application discloses a device processor applied to an audio-video synchronous acquisition system, the audio-video synchronous acquisition system comprising a handheld device main body, an audio acquisition module and a video acquisition module, the audio acquisition module and the video acquisition module being arranged on the handheld device main body, and the device processor being integrated in the handheld device main body, The method comprises the following steps: in response to a synchronous acquisition starting instruction, simultaneously starting the audio acquisition module and the video acquisition module; recording audio data emitted by a test object through the audio acquisition module and recording a face video of the test object through the video acquisition module; synchronously processing the audio data and the face video; saving the audio data and the face video after synchronous processing.

2. The method of claim 1, wherein, The audio data and the face video are synchronously processed in the following manner: performing image processing on the face video to generate a mouth movement video of the test object; adding a uniform timestamp to the mouth movement video and the face video.

3. The method of claim 2, wherein, The face video comprises a plurality of face video frames, The mouth movement video is generated in the following manner: for each face video frame, performing the following steps: extracting a mouth feature image of the test object from the face video frame, taking the mouth feature image as a mouth feature video frame corresponding to the face video frame, and replacing the face video frame with the mouth feature video frame; generating a mouth movement video of the test object from the mouth feature video frames.

4. The method of claim 2, wherein, The mouth movement video comprises a plurality of mouth feature video frames, and the audio data comprises a plurality of audio frames, The step of adding a uniform timestamp to the mouth movement video and the face video comprises the following steps: adding a same timestamp to a mouth feature video frame and an audio frame at a same acquisition time.

5. The method of claim 1, wherein, The audio-video synchronous acquisition system further comprises a data processing terminal, the data processing terminal being connected with the device processor through a preset communication mode, The method further comprises the following steps: receiving a data reading instruction transmitted by the data processing terminal; in response to the data reading instruction, transmitting the audio data and the face video after synchronous processing to the data processing terminal.

6. The method of claim 1, wherein, The audio acquisition module and the video acquisition module are arranged in close proximity.

7. The method of claim 1, wherein, An optical axis of the video acquisition module is parallel to an optimal sound pickup axis of the audio acquisition module.

8. A speech ability data collection apparatus characterized by comprising: The device processor is applied to an audio-video synchronous acquisition system, the audio-video synchronous acquisition system comprising a handheld device main body, an audio acquisition module and a video acquisition module, the audio acquisition module and the video acquisition module being arranged on the handheld device main body, and the device processor being integrated in the device main body, The device comprises the following modules: an instruction response module, configured to simultaneously start the audio acquisition module and the video acquisition module in response to a synchronous acquisition starting instruction; an acquisition module, configured to record audio data emitted by a test object through the audio acquisition module and record a face video of the test object through the video acquisition module; a synchronous module, configured to synchronously process the audio data and the face video; a saving module, configured to save the audio data and the face video after synchronous processing.

9. An audio-video synchronization acquisition system, characterized in that, The audio-video synchronous acquisition system comprises a handheld device main body, an audio acquisition module and a video acquisition module, the audio acquisition module and the video acquisition module are arranged on the handheld device main body, a device processor is integrated in the device main body, and the device processor runs the speech ability data acquisition method as claimed in any one of claims 1-7.

10. A data processing terminal, characterized by The data processing terminal sends a data reading instruction to the device processor in the audio-video synchronous acquisition system as claimed in any one of claims 1-7. The audio data and the facial video processed synchronously and fed back by the device processor are received and saved.

Images