Audio and video recording method, wearable device, terminal device and readable storage medium

Abstract

The application discloses an audio and video recording method, a wearable device, a terminal device and a readable storage medium, relates to the technical field of wearable devices, and applies to the wearable device. The wearable device is in communication connection with the terminal device. The wearable device comprises a microcontroller unit (MCU) and an image signal processor (ISP). The method comprises the following steps: in response to an input audio and video recording instruction, recording an audio file through the MCU and recording a video file through the ISP; determining time synchronization information between the audio file and the video file; and sending the audio file, the video file and target synchronization information to the terminal device, so that the terminal device performs audio and video synthesis on the audio file and the video file based on the target synchronization information, and obtains an audio and video file. The application can reduce the demand of the audio and video recording function on the computing power resources of the wearable device.

Description

Technical Field

[0001] This application relates to the field of wearable device technology, and more particularly to audio and video recording methods, wearable devices, terminal devices, and readable storage media. Background Technology

[0002] With the development of lightweight and miniaturized wearable devices (such as smartwatches and smart glasses), their internal space and hardware configuration are strictly limited, resulting in a low level of computing resources.

[0003] However, audio and video recording is an important function of wearable devices. The recording process requires continuous audio and video recording and audio-video synthesis, which places high demands on the computing resources of wearable devices.

[0004] Therefore, in the pursuit of lightweight and miniaturized wearable devices, how to effectively address the high demand for computing resources in audio and video recording functions has become an urgent technical challenge. Summary of the Invention

[0005] The main purpose of this application is to provide an audio and video recording method, which aims to solve the technical problem that the computing power required for audio and video recording functions is mismatched with the actual computing power available for audio and video recording functions in wearable devices.

[0006] To achieve the above objectives, this application provides an audio and video recording method, which is applied to a wearable device. The wearable device is communicatively connected to a terminal device, and includes a microcontroller unit (MCU) and an image signal processor (ISP). The method includes:

[0007] In response to the input audio and video recording command, an audio file is recorded by the MCU, and a video file is recorded by the ISP;

[0008] Determine the time synchronization information between the audio file and the video file;

[0009] The audio file, the video file, and the target synchronization information are sent to the terminal device, so that the terminal device can perform audio-video synthesis on the audio file and the video file based on the target synchronization information to obtain an audio-video file. The target synchronization information is the time synchronization information, or it is file alignment information obtained by aligning the audio file and the video file in time based on the time synchronization information.

[0010] In one embodiment, the step of determining the time synchronization information between the audio file and the video file includes:

[0011] Determine the single signal transmission time between the MCU and the ISP, and determine the time offset between the audio file and the video file based on the single signal transmission time;

[0012] The time offset is used as the time synchronization information between the audio file and the video file.

[0013] In one embodiment, the step of determining the time synchronization information between the audio file and the video file includes:

[0014] Obtain the first local timestamp information corresponding to the audio file and the second local timestamp information corresponding to the video file;

[0015] The first local timestamp information and the second local timestamp information are used as time synchronization information between the audio file and the video file.

[0016] In one embodiment, the step of determining the time synchronization information between the audio file and the video file includes:

[0017] Obtain the MCU timestamp corresponding to the audio file, the ISP timestamp corresponding to the video file, and the timestamp mapping relationship between the MCU and the ISP;

[0018] The MCU timestamp, the ISP timestamp, and the timestamp mapping relationship are used as time synchronization information between the audio file and the video file.

[0019] In one embodiment, the audio and video recording instructions include a start recording instruction and a stop recording instruction. The step of recording target audio through the MCU and recording target video through the ISP in response to the input audio and video recording instructions includes:

[0020] In response to the input start recording command, the MCU is triggered to start audio recording, and the MCU is also triggered to send a start recording signal to the ISP, wherein the start recording signal is used to trigger the ISP to start video recording;

[0021] In response to the input end recording command, the MCU is triggered to send an end recording signal to the ISP to obtain the recorded video file, and the MCU is triggered to end audio recording after receiving the recording end signal returned by the ISP to obtain the recorded audio file. The end recording signal is used to trigger the ISP to end video recording and return the recording end signal to the MCU after ending video recording.

[0022] In one embodiment, the wearable device further includes an inertial measurement unit (IMU), and the method further includes:

[0023] Acquire the IMU data collected by the IMU during the recording of the video file;

[0024] The IMU data is sent to the terminal device so that the terminal device performs electronic image stabilization on the audio and video files based on the IMU data.

[0025] Furthermore, to achieve the above objectives, this application also provides an audio and video recording method, characterized in that the audio and video recording method is applied to a terminal device, the terminal device being communicatively connected to a wearable device, the wearable device including a microcontroller unit (MCU) and an image signal processor (ISP), and the method comprising:

[0026] Obtain the audio files, video files, and target synchronization information sent by the wearable device;

[0027] Based on the target synchronization information, the audio file and the video file are combined to obtain an audio-video file;

[0028] The audio file is recorded by the wearable device in response to an input audio / video recording command through the MCU, and the video file is recorded by the wearable device in response to an input audio / video recording command through the ISP. The target synchronization information is the time synchronization information between the audio file and the video file, or it is the file alignment information obtained by aligning the audio file and the video file in time based on the time synchronization information.

[0029] In one embodiment, the step of synthesizing the audio file and the video file based on the target synchronization information to obtain an audio-video file includes:

[0030] When the target synchronization information is time synchronization information, the audio file and the video file are time-aligned based on the time synchronization information to obtain file alignment information, and the audio file and the video file are then combined according to the file alignment information to obtain an audio-video file; or,

[0031] When the target synchronization information is file alignment information, the audio file and the video file are combined according to the file alignment information to obtain an audio-video file.

[0032] In one embodiment, the time synchronization information includes at least one of the following:

[0033] The time offset between the audio file and the video file, wherein the time offset is determined based on the single signal transmission time between the MCU and the ISP;

[0034] The first local timestamp information corresponding to the audio file and the second local timestamp information corresponding to the video file;

[0035] The MCU timestamp corresponding to the audio file, the ISP timestamp corresponding to the video file, and the timestamp mapping relationship between the MCU and the ISP.

[0036] In one embodiment, the wearable device further includes an inertial measurement unit (IMU), and the method further includes:

[0037] Acquire IMU data sent by the wearable device, wherein the IMU data is data collected by the IMU during the recording of the video file;

[0038] Based on the IMU data, electronic image stabilization is applied to the audio and video files.

[0039] In addition, to achieve the above objectives, this application also provides a wearable device, the wearable device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the audio and video recording method described above.

[0040] In addition, to achieve the above objectives, this application also provides a terminal device, the terminal device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the audio and video recording method described above.

[0041] In addition, to achieve the above objectives, this application also provides a readable storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the audio and video recording method described above.

[0042] This application provides an audio and video recording method, a wearable device, a terminal device, and a readable storage medium. The audio and video recording method is applied to a wearable device, which is communicatively connected to the terminal device. The wearable device includes a microcontroller unit (MCU) and an image signal processor (ISP). The method includes: responding to an input audio and video recording command, recording an audio file through the MCU and a video file through the ISP; determining time synchronization information between the audio file and the video file; and sending the audio file, the video file, and target synchronization information to the terminal device, so that the terminal device performs audio and video synthesis based on the target synchronization information to obtain an audio and video file. The target synchronization information is either time synchronization information or file alignment information obtained by time-aligning the audio file and the video file based on the time synchronization information.

[0043] This application embodiment breaks down the audio and video recording task into three parts: audio recording, video recording, and audio and video synthesis. Audio recording is completed on the MCU side of the wearable device, and video recording is completed on the ISP side. The high-computing-power audio and video synthesis is transferred from the wearable device with limited computing resources to the terminal device (such as mobile devices like mobile phones and tablets) used by the user to actually view audio and video files. This avoids performing complex audio and video synthesis tasks on wearable devices with limited computing resources, and prevents excessive computing power consumption caused by the wearable device performing high-computing-power audio and video synthesis tasks locally. It significantly reduces the operating load and power consumption of the wearable device, making it easier to achieve lightweight and miniaturized wearable devices. At the same time, by calculating and sending target synchronization information, it ensures that the terminal device can achieve high-precision audio and video synchronization when performing audio and video synthesis. Thus, it effectively solves the technical problem of the mismatch between the computing power requirements of the audio and video recording function and the actual computing power available for audio and video recording in the wearable device. Attached Figure Description

[0044] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0045] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0046] Figure 1 This is a flowchart illustrating the first embodiment of the audio / video recording method of this application.

[0047] Figure 2 This is a flowchart illustrating the second embodiment of the audio / video recording method of this application.

[0048] Figure 3 This is a flowchart illustrating the third embodiment of the audio / video recording method of this application.

[0049] Figure 4 This is a flowchart illustrating the fourth embodiment of the audio / video recording method of this application.

[0050] Figure 5 This is a schematic diagram of the system framework in a specific embodiment of this application;

[0051] Figure 6 This is a schematic diagram of the audio and video recording business process in a specific embodiment of this application;

[0052] Figure 7 This is a schematic diagram of timestamp synchronization in a specific embodiment of this application;

[0053] Figure 8 This is a schematic diagram of the hardware operating environment of the wearable device involved in the audio and video recording method in the embodiments of this application;

[0054] Figure 9 This is a schematic diagram of the hardware operating environment of the terminal device involved in the audio and video recording method in this application embodiment.

[0055] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0056] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0057] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0058] Currently, in related technologies, audio and video recording of wearable devices is usually completed on the wearable device itself, including audio recording, video recording, and audio-video synthesis. Finally, the audio and video files that are synchronized with the video are sent to the terminal device for users to view on the terminal device.

[0059] In the context of miniaturization and lightweight development of wearable devices, the limited battery capacity, memory capacity, and computing resources of these devices make it difficult to handle tasks with high computing power requirements (tasks with high computing power requirements consume more power and have higher memory usage). However, audio and video recording (especially audio and video synthesis) has high requirements for power consumption, memory, and computing power, resulting in a mismatch between the resource requirements of audio and video recording functions and the actual resource situation of wearable devices.

[0060] The main solution of this application embodiment is an audio and video recording method applied to a wearable device. The wearable device is communicatively connected to a terminal device, and includes a microcontroller unit (MCU) and an image signal processor (ISP). The method includes: responding to an input audio and video recording command, recording an audio file through the MCU and a video file through the ISP; determining time synchronization information between the audio file and the video file; and sending the audio file, the video file, and target synchronization information to the terminal device, so that the terminal device performs audio and video synthesis on the audio file and the video file based on the target synchronization information to obtain an audio and video file. The target synchronization information is either the time synchronization information or file alignment information obtained by time-aligning the audio file and the video file based on the time synchronization information.

[0061] This application embodiment breaks down the audio and video recording task into three parts: audio recording, video recording, and audio and video synthesis. Audio recording is completed on the MCU side of the wearable device, and video recording is completed on the ISP side. The high-computing-power audio and video synthesis is transferred from the wearable device with limited computing resources to the terminal device (such as mobile devices like mobile phones and tablets) used by the user to actually view audio and video files. This avoids performing complex audio and video synthesis tasks on wearable devices with limited computing resources, and prevents excessive computing power consumption caused by the wearable device performing high-computing-power audio and video synthesis tasks locally. It significantly reduces the operating load and power consumption of the wearable device, making it easier to achieve lightweight and miniaturized wearable devices. At the same time, by calculating and sending target synchronization information, it ensures that the terminal device can achieve high-precision audio and video synchronization when performing audio and video synthesis. Thus, it effectively solves the technical problem of the mismatch between the computing power requirements of the audio and video recording function and the actual computing power available for audio and video recording in the wearable device.

[0062] It should be noted that the execution subject of the embodiments of this application is a wearable device, which may include, but is not limited to, smart glasses, smartwatches, smart helmets, or any electronic device capable of realizing the above functions. The following description uses a wearable device as the execution subject to illustrate the various embodiments of this application.

[0063] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0064] This application proposes an audio and video recording method according to a first embodiment.

[0065] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the audio / video recording method of this application.

[0066] In this embodiment, the audio and video recording method is applied to a wearable device, which is communicatively connected to a terminal device. The wearable device includes a microcontroller unit (MCU) and an image signal processor (ISP). The method includes steps S100 to S300:

[0067] In step S100, in response to the input audio and video recording command, an audio file is recorded by the MCU and a video file is recorded by the ISP.

[0068] As those skilled in the art will know, an MCU (Microcontroller Unit) is a microcomputer system that integrates a processor core, memory, input / output peripherals, and timers, and is typically embedded in other devices to perform specific tasks. An ISP (Image Signal Processor) is a hardware or software module specifically designed for processing image signals. It is widely used in devices such as digital cameras, smartphones, surveillance cameras, and automotive imaging systems. Its main function is to process and optimize raw data from image sensors to generate high-quality image or video output.

[0069] In this embodiment, the MCU is mainly responsible for optimizing and encoding the audio data captured by the microphone to generate high-quality audio files, while the ISP is mainly responsible for optimizing and encoding the image data captured by the camera to generate high-quality video files.

[0070] It should be noted that the audio and video recording command is an operation command used to control the audio and video recording function of the wearable device. This operation command can be input through physical buttons, touch screen, voice control or other interactive methods to instruct the wearable device to record audio and video. This embodiment does not make specific limitations on this.

[0071] In this embodiment, audio and video recording is split into audio recording, video recording, and audio-video synthesis. Audio recording is completed on the MCU side, and video recording is completed on the ISP side. This allows the two to execute the independent audio and video recording processes in parallel. By utilizing the low power consumption of the MCU and the parallel processing capabilities of the ISP, resource allocation is optimized. This avoids the context switching overhead caused by the processor core frequently switching between audio encoding / decoding and image processing tasks when a single processor is responsible for both audio and video recording in traditional solutions. It significantly reduces the risk of memory bus contention in wearable devices, enabling audio and video files to be generated in parallel and with high quality.

[0072] Step S200: Determine the time synchronization information between the audio file and the video file;

[0073] It should be noted that time synchronization information is used to establish the correspondence between audio files and video files on the timeline.

[0074] This embodiment ensures that when a terminal device performs audio and video synthesis, it can accurately align the audio and video files on the timeline based on the time synchronization information by determining the time synchronization information between the audio and video files. This avoids audio-visual asynchrony caused by internal delays or clock differences in wearable devices and lays the foundation for high-quality audio-visual synthesis.

[0075] Step S300: The audio file, video file, and target synchronization information are sent to the terminal device so that the terminal device can synthesize the audio file and video file based on the target synchronization information to obtain an audio and video file. The target synchronization information is time synchronization information, or file alignment information obtained by aligning the audio file and video file based on the time synchronization information.

[0076] It's important to note that target synchronization information guides terminal devices on how to align audio and video files on the timeline. Time alignment refers to the process of establishing the correspondence between audio and video files on the timeline. File alignment information describes this correspondence, ensuring precise alignment of audio and video on the timeline, thus achieving a high degree of consistency between sound and picture when playing audio and video files.

[0077] As those skilled in the art will understand, audio-visual synthesis refers to the process of integrating independently recorded audio and video files to generate a complete audio-visual file containing synchronized audio and video content. Specifically, this involves audio and video file encoding and decoding, as well as related algorithms for audio-visual synthesis (such as timestamp synchronization, audio file preprocessing algorithms, video file algorithms, and format conversion). This process demands significant computing resources, resulting in a high load and power consumption for increasingly lightweight wearable devices. The algorithms related to audio-visual synthesis have been extensively studied by those skilled in the art and will not be elaborated upon here.

[0078] In this embodiment, the wearable device records audio and video files through the MCU and ISP respectively. After determining the time synchronization information between the audio and video files, it can directly use the time synchronization information as the target synchronization information and send it to the terminal device along with the audio and video files. Alternatively, it can first perform time alignment on the audio and video files based on the time synchronization information to establish the correspondence between the audio and video files on the timeline, obtain file alignment information, and then use the file alignment information as the target synchronization information and send it to the terminal device along with the audio and video files.

[0079] It's easy to understand that when a wearable device directly sends this time synchronization information as the target synchronization information, along with audio and video files, to the terminal device, the terminal device first needs to perform time alignment on the audio and video files based on this target synchronization information. This establishes the correspondence between the audio and video files on the timeline, obtaining file alignment information. Only then can the audio and video files be synthesized based on this file alignment information to obtain the final audio and video file. Preferably, the time alignment of audio and video files requires certain computing resources. Therefore, delegating this time alignment step to the terminal device not only fully utilizes the terminal device's stronger processing power but also reduces the workload of the wearable device, extends its battery life, and reduces issues such as overheating.

[0080] When a wearable device first aligns the audio and video files based on the time synchronization information, establishing a correspondence between them on the timeline and obtaining file alignment information, and then sends this alignment information, along with the audio and video files, to the terminal device as target synchronization information, the terminal device can directly synthesize the audio and video files based on this target synchronization information to obtain the final audio and video file. Since the computational resources required for time alignment are far less than those for audio and video synthesis, falling within the capabilities of wearable devices, delegating this step to the wearable device not only avoids consuming excessive computational resources but also eliminates the need for additional time calibration steps. This allows the terminal device to focus more on core tasks of audio and video synthesis, such as encoding conversion and format encapsulation, thus improving overall processing efficiency.

[0081] This embodiment breaks down the audio and video recording task into three parts: audio recording, video recording, and audio-video synthesis. Audio recording is completed on the MCU side of the wearable device, and video recording is completed on the ISP side. The computationally intensive audio and video synthesis is transferred from the wearable device with limited computing resources to the terminal device (such as a mobile phone or tablet) used by the user to view the audio and video files. This avoids performing complex audio and video synthesis tasks on wearable devices with limited computing resources, preventing excessive computing power consumption caused by the wearable device performing high-computation audio and video synthesis tasks locally. This significantly reduces the operating load and power consumption of the wearable device, facilitating the lightweighting and miniaturization of the wearable device. At the same time, by calculating and sending target synchronization information, it ensures that the terminal device can achieve high-precision audio and video synchronization when performing audio and video synthesis. This effectively solves the technical problem of the mismatch between the computing power requirements of the audio and video recording function and the actual computing power available for audio and video recording on the wearable device.

[0082] It's worth noting that in practical applications, users typically don't preview audio and video files directly on wearable devices. Instead, they browse the recorded content on mobile phones, computers, and other terminal devices. Therefore, compared to performing audio and video synthesis on wearable devices with limited computing resources and transmitting audio and video files to the terminal device, performing audio and video synthesis on the terminal device can significantly reduce the computing resource requirements of the audio and video recording function on the wearable device without affecting the user's browsing of audio and video files. This effectively alleviates the burden on the wearable device, allowing it to focus its limited computing resources on audio and video acquisition and encoding, ensuring the stability and efficiency of the recording process, avoiding stuttering or data loss due to insufficient resources, and fully utilizing the powerful performance of the terminal device to efficiently complete the audio and video synthesis task. It also supports higher quality output (such as high resolution and high frame rate), improving the quality of the final audio and video files and providing users with a better browsing experience.

[0083] In a first feasible implementation, step S200 may include steps S210 to S220:

[0084] Step S210: Determine the single signal transmission time between the MCU and the ISP, and determine the time offset between the audio file and the video file based on the single signal transmission time;

[0085] Step S220: Use the time offset as time synchronization information between the audio file and the video file.

[0086] It should be noted that the single signal transmission time refers to the time required for a signal to travel from the MCU to the ISP, or from the ISP to the MCU, within the wearable device. This time can be measured by pre-embedding test signals or recording the handshake protocol during the initialization phase. The time offset refers to the deviation between the audio and video files in terms of their start time.

[0087] It's easy to understand that, since the MCU and ISP in wearable devices are independent modules, when responding to audio and video recording commands, the MCU needs to send a start recording signal to notify the ISP to start video recording. However, there is a delay in signal transmission between the MCU and the ISP. Therefore, the recorded audio and video files will have a discrepancy in their start time.

[0088] In this embodiment, after measuring the single signal transmission time between the MCU and the ISP, the deviation between the audio file and the video file at the start time can be quantified using this single signal transmission time. The resulting time offset is then used as the time synchronization information between the audio file and the video file, so as to establish the correspondence between the audio file and the video file on the time axis, eliminate the deviation at the start time, and facilitate compensation in the subsequent time alignment process of the video file and the audio file. This compensation method can align the start time points of the audio and video, avoid audio-visual asynchrony problems caused by internal delays in wearable devices, and ensure that the final synthesized audio and video files have high consistency and smoothness.

[0089] Furthermore, this implementation method is simple and direct, reduces additional data processing steps, and improves efficiency.

[0090] In a second feasible implementation, step S200 may include steps S230 to S240:

[0091] Step S230: Obtain the first local timestamp information corresponding to the audio file and the second local timestamp information corresponding to the video file;

[0092] Step S240: Use the first local timestamp information and the second local timestamp information as time synchronization information between the audio file and the video file.

[0093] It should be noted that in this embodiment, the MCU and ISP use a unified clock source.

[0094] It should also be noted that a local timestamp refers to the point in time when an audio or video frame is generated, under the same unified clock source. The first local timestamp refers to the local timestamp marked on the audio frame when it is generated in an audio file, and the second local timestamp refers to the local timestamp marked on the video frame when it is generated in a video file.

[0095] In this embodiment, with a unified clock source, the first local timestamp information corresponding to the audio file and the second local timestamp information corresponding to the video file are used as time synchronization information between the audio file and the video file. This intuitively reflects the recording time order between the audio file and the video file, making it easier to align the audio file and the video file in time and ensuring that the synthesized audio and video file has good synchronization.

[0096] In a third feasible implementation, step S200 may include steps S250 to S260:

[0097] Step S250: Obtain the MCU timestamp corresponding to the audio file, the ISP timestamp corresponding to the video file, and the timestamp mapping relationship between the MCU and the ISP;

[0098] Step S260: Use the MCU timestamp, ISP timestamp, and timestamp mapping relationship as time synchronization information between the audio file and the video file.

[0099] It should be noted that in this embodiment, the MCU and ISP each use independent clock sources.

[0100] It should also be noted that the MCU timestamp refers to the time point when an audio frame is generated under the independent clock source corresponding to the MCU, while the ISP timestamp refers to the time point when a video frame is generated under the independent clock source corresponding to the ISP. The timestamp mapping relationship between MCU and ISP refers to the correspondence between the MCU timestamp and the ISP timestamp.

[0101] It is easy to understand that the timestamp mapping relationship can be pre-defined or real-time defined, and this implementation does not impose any specific restrictions on this.

[0102] In this embodiment, when clock sources are heterogeneous, the MCU timestamp corresponding to the audio file, the ISP timestamp corresponding to the video file, and the timestamp mapping relationship between the MCU and the ISP are used as time synchronization information between the audio and video files. Through this timestamp mapping relationship, the MCU timestamp and the ISP timestamp can be unified to the same time coordinate system, solving the timing incomparability problem caused by clock source heterogeneity. This facilitates accurate time alignment of the audio and video files, ensuring that the final synthesized audio and video files have extremely high synchronization and quality.

[0103] Based on the first embodiment described above, a second embodiment of the audio and video recording method of this application is proposed.

[0104] In the second embodiment of this application, the same or similar content as in the above embodiments can be referred to the above description, and will not be repeated hereafter.

[0105] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating the second embodiment of the audio / video recording method of this application.

[0106] In this embodiment, the audio and video recording command includes a start recording command and a stop recording command, and step S100 may include steps S110 to S120:

[0107] Step S110: In response to the input start recording command, the MCU is triggered to start audio recording and to send a start recording signal to the ISP. The start recording signal is used to trigger the ISP to start video recording.

[0108] In step S120, in response to the input end recording command, the MCU is triggered to send an end recording signal to the ISP to obtain the recorded video file, and the MCU is triggered to end audio recording after receiving the recording end signal returned by the ISP to obtain the recorded audio file. The end recording signal is used to trigger the ISP to end video recording and return a recording end signal to the MCU after ending video recording.

[0109] It should be noted that, in this embodiment, the audio and video recording instructions include a start recording instruction and an end recording instruction. The start recording instruction is an operation instruction used to enable the audio and video recording function of the wearable device, instructing the wearable device to start recording audio and video. The end recording instruction is an operation instruction used to disable the audio and video recording function of the wearable device, instructing the wearable device to stop recording audio and video.

[0110] It is understood that both operation commands can be input via physical buttons, touch screen, voice control, or other interactive methods, and this embodiment does not impose specific limitations on them.

[0111] This embodiment, by introducing explicit start and end recording commands and corresponding signal transmission mechanisms between the MCU and ISP, enables more precise control over the start and end points of audio and video recording, avoiding unnecessary data acquisition and reducing resource waste. Furthermore, this design allows for effective communication between the MCU and ISP, ensuring that audio and video recording starts and stops accurately at predetermined times, thereby improving the overall system stability and reliability.

[0112] Based on the above embodiments, a third embodiment of the audio and video recording method of this application is proposed.

[0113] In the third embodiment of this application, the same or similar content as the above embodiments can be referred to the above description, and will not be repeated hereafter.

[0114] Please refer to Figure 3 , Figure 3 This is a flowchart illustrating the third embodiment of the audio / video recording method of this application.

[0115] In this embodiment, the wearable device further includes an inertial measurement unit (IMU), and the audio and video recording method may further include steps S400 to S500:

[0116] Step S400: Obtain IMU data acquired by the IMU during the recording of the video file;

[0117] Step S500: IMU data is sent to the terminal device so that the terminal device can perform electronic image stabilization on the audio and video files based on the IMU data.

[0118] As those skilled in the art will know, an IMU (Inertial Measurement Unit) is a sensor used to measure the motion state and spatial attitude of an object, typically including an accelerometer and a gyroscope. IMU data refers to the data measured by the IMU, which usually includes linear acceleration data and angular velocity data.

[0119] It should be noted that electronic image stabilization refers to dynamically cropping or deforming video footage to counteract the effects of shaking by analyzing the correspondence between motion data from wearable devices (especially image sensors in wearable devices) and video frame sequences.

[0120] In this embodiment, during the recording of video files, the IMU accurately captures the motion trajectory of the wearable device (especially the image sensor in the wearable device) while recording video, and transmits the acquired IMU data to the terminal device. This enables the terminal device to identify and compensate for the image jitter caused by the irregular movement of the wearable device based on the IMU data, thereby improving the stability of the final output audio and video files and the viewing experience.

[0121] Furthermore, this embodiment transfers electronic image stabilization processing from wearable devices with limited computing resources to terminal devices, making full use of the powerful computing resources of terminal devices while maintaining the lightweight and high-efficiency characteristics of wearable devices, ensuring that users can enjoy high-quality audio and video recording while obtaining a smoother and more comfortable visual experience.

[0122] This application proposes a fourth embodiment of an audio and video recording method.

[0123] In the fourth embodiment of this application, the same or similar content as in the above embodiments can be referred to the above description, and will not be repeated hereafter.

[0124] Please refer to Figure 4 , Figure 4 This is a flowchart illustrating the fourth embodiment of the audio / video recording method of this application.

[0125] In this embodiment, the audio and video recording method is applied to a terminal device, which is communicatively connected to a wearable device. The wearable device includes a microcontroller unit (MCU) and an image signal processor (ISP). The method includes steps S600 to S700:

[0126] Step S600: Obtain the audio files, video files, and target synchronization information sent by the wearable device;

[0127] Step S700: Based on the target synchronization information, the audio file and video file are combined to obtain an audio and video file.

[0128] The audio file is recorded by the MCU in response to the input audio and video recording command of the wearable device. The video file is recorded by the ISP in response to the input audio and video recording command of the wearable device. The target synchronization information is the time synchronization information between the audio file and the video file, or the file alignment information obtained by aligning the audio file and the video file based on the time synchronization information.

[0129] In this embodiment, the terminal device may include, but is not limited to, mobile terminals such as smartphones, laptops, PADs (Portable Application Description, tablet computers), and fixed terminals such as desktop computers, or any electronic device capable of performing the above functions. This application embodiment does not specifically limit this.

[0130] In one possible implementation, step S700 may include step S710 or step S720:

[0131] Step S710: If the target synchronization information is time synchronization information, perform time alignment on the audio file and video file based on the time synchronization information to obtain file alignment information, and perform audio-video synthesis on the audio file and video file according to the file alignment information to obtain an audio-video file.

[0132] In this embodiment, when the target synchronization information is time synchronization information, the terminal device first needs to perform time alignment on the audio and video files based on the target synchronization information to establish the correspondence between the audio and video files on the timeline and obtain file alignment information. Only then can the audio and video files be synthesized according to the file alignment information to obtain the audio and video file. Since the time alignment of audio and video files requires certain computing resources, delegating this step to the terminal device can not only make full use of the terminal device's stronger processing power, but also reduce the workload of wearable devices, extend their battery life, and reduce heat generation and other issues.

[0133] Alternatively, in step S720, if the target synchronization information is file alignment information, the audio file and video file are synthesized according to the file alignment information to obtain an audio and video file.

[0134] In this embodiment, when the target synchronization information is file alignment information, the terminal device can directly synthesize audio and video files based on the target synchronization information to obtain an audio and video file. Since the computing resources required for time alignment are far less than those required for audio and video synthesis, and are generally within the capacity of wearable devices, delegating the time alignment step to the wearable device not only avoids consuming excessive computing resources of the wearable device, but also eliminates the need for additional time calibration steps, allowing the terminal device to focus more on the core tasks of audio and video synthesis, such as encoding conversion and format encapsulation, thereby improving overall processing efficiency.

[0135] In one feasible implementation, the time synchronization information includes at least one of the following:

[0136] The time offset between audio and video files, where the time offset is determined based on the single signal transmission time between the MCU and the ISP;

[0137] The first local timestamp information corresponding to the audio file and the second local timestamp information corresponding to the video file;

[0138] The MCU timestamp corresponding to the audio file, the ISP timestamp corresponding to the video file, and the timestamp mapping relationship between the MCU and the ISP.

[0139] In one feasible implementation, the wearable device further includes an inertial measurement unit (IMU), and the audio / video recording method may further include steps S800 to S900:

[0140] Step S800: Obtain IMU data sent by the wearable device, wherein the IMU data is the data collected by the IMU during the recording of the video file;

[0141] Step S900: Perform electronic image stabilization on the audio and video files based on IMU data.

[0142] The beneficial effects of the audio and video recording method provided in this embodiment are the same as those of the audio and video recording method provided in the above embodiments. It can solve the technical problem that the demand for computing power resources for audio and video recording functions is not matched with the actual computing power resources available for audio and video recording functions in wearable devices, and will not be elaborated here.

[0143] To facilitate understanding of the technical concept or principle of the audio and video recording method of this application as described in the above embodiments, a specific embodiment is given below:

[0144] like Figure 5As shown in this specific embodiment, the wearable device mainly includes an audio module, a video module, an MCU, and an ISP. The audio module is used to acquire audio signals, the camera module is used to acquire image signals, the ISP acquires images through the camera module, and performs video encoding on the acquired image signals to obtain a recorded video file. Finally, the recorded video file is transmitted to the MCU via SPI (Serial Peripheral Interface). The MCU acquires audio through the audio module and performs audio encoding on the acquired audio signals to obtain a recorded audio file. After recording the audio and video files, the MCU is responsible for file storage and forwarding. The MCU communicates with the mobile phone via BT (Bluetooth), WIFI (Wireless Fidelity), or USB (Universal Serial Bus) to send the recorded audio and video files to the mobile phone (i.e., the smart terminal). Finally, the audio and video are decoded on the mobile phone, and the decoded audio and video are synthesized.

[0145] like Figure 6 As shown, in this specific embodiment, smart glasses are used as wearable devices and mobile phones are used as terminal devices. The smart glasses include a glasses MCU and a glasses ISP, and the smart glasses and mobile phones are connected via WIFI communication.

[0146] During the command input phase, users can send audio and video recording commands in various ways to trigger the smart glasses to record audio and video. For example, they can be triggered via Bluetooth commands, AI (Artificial Intelligence) triggers, or voice triggers on a mobile application, or via the Camera button on the smart glasses.

[0147] During the device response phase, the smart glasses receive audio and video recording commands through the recording interface, triggering the glasses' MCU to record audio and encapsulate the recording ISP command, which is then sent to the glasses' ISP via SPI. This triggers the glasses' ISP to record video and the IMU to acquire motion data (i.e., to obtain the IMU data acquired during the recording of the video file). Finally, the glasses obtain AAC (Advanced Audio Coding) files (i.e., audio files), MP4 files (i.e., video files), and IMUTXT (Text File) files (i.e., IMU data), which are then sent to the mobile application via Wi-Fi.

[0148] During the terminal processing stage, after the mobile application receives the file sent by the smart glasses via WIFI SYNC (Wireless Fidelity Synchronization), it first performs audio decoding on the AAC file and video decoding on the MP4 file through video processing. Then, it combines the decoded AAC file and MP4 file into an audio and video file through A / V (Audio / Video) synthesis. Next, it performs EIS (Electronic Image Stabilization) processing on the audio and video file through IMU TXT file, and finally outputs an electronically stabilized MP4 format audio and video file (that is, electronic image stabilization is performed on the audio and video file based on IMU data).

[0149] In this specific embodiment, the audio and video recording process is divided into recording preparation, recording start, recording end, file transfer, and terminal processing.

[0150] During the recording preparation phase, it is necessary to synchronize the timestamps of the MCU and ISP, such as... Figure 7 As shown, the MCU sends a synchronization command value (ISP). The synchronization command data structure is: SPI basic data format + synchronization command ID + current MCU time A. After receiving the synchronization command, the ISP performs timestamp synchronization. After synchronization is complete, it sends a return command to the MCU. The return command data structure is: SPI basic data format + return command ID + current ISP time B. After receiving the return command, the MCU obtains the current MCU time C, and then uses the following method to obtain the approximate transmission time of the ISP command (i.e., the single signal transmission time) D = ((CA) - (BA)) / 2, that is, D = (CB) / 2, which is used for compensation during subsequent audio and video synthesis. That is, the single signal transmission time between the MCU and the ISP is determined, and the time offset between the audio file and the video file is determined based on the single signal transmission time; the time offset is used as the time synchronization information between the audio file and the video file.

[0151] During the recording start phase, the start recording command can be input in various ways, such as via buttons on the glasses, AI control, voice control, or commands from a mobile app. When recording begins, the MCU sends a start recording command (i.e., a start recording signal) to the ISP. The command format is: SPI data basic format + recording command ID + video parameters + filename. Upon receiving the start recording command, the ISP begins recording video. Then, the MCU simultaneously begins audio recording, maintaining a name association between the audio and video files. In other words, in response to the input start recording command, the MCU is triggered to begin audio recording and sends a start recording signal to the ISP, which in turn triggers the ISP to begin video recording.

[0152] During the recording termination phase, the end-of-recording command can be input in various ways, such as via buttons on the glasses, AI control, voice control, mobile app commands, or the expiration of a pre-set timeout. When recording ends, the MCU sends an end-of-recording command (i.e., an end-of-recording signal) to the ISP and waits for a return end-of-recording signal from the ISP. Upon receiving the end-of-recording command, the ISP stops recording, saves the video file, and sends an end-of-recording signal back to the MCU, indicating successful termination. Upon receiving the end-of-recording signal from the ISP, the MCU stops recording audio and saves the audio file. In other words, in response to the input end-of-recording command, the MCU sends an end-of-recording signal to the ISP, obtaining the recorded video file, and upon receiving the end-of-recording signal from the ISP, the MCU stops audio recording, obtaining the recorded audio file. The end-of-recording signal triggers the ISP to stop video recording and returns an end-of-recording signal to the MCU after video recording ends.

[0153] During the file transfer phase, the wearable device transmits audio files, video files, and ISP commands (approximately time D) to the mobile application via Bluetooth or Wi-Fi. In other words, it sends audio files, video files, and target synchronization information to the terminal device.

[0154] During the terminal processing stage, the mobile application first decodes the audio and video files, and then, based on the approximate transmission time D according to the ISP command, merges the audio and video files into a single audio / video file. In other words, based on the target synchronization information, the audio and video files are combined to obtain the final audio / video file.

[0155] It should be noted that the above examples are only for the purpose of assisting in understanding this application and do not constitute a limitation on the audio and video recording method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0156] In addition, please refer to Figure 8 , Figure 8 This is a schematic diagram of the hardware operating environment of the wearable device involved in the audio and video recording method in the embodiments of this application.

[0157] This application also provides a wearable device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the steps of the audio and video recording method in the above embodiments.

[0158] The following is for reference. Figure 8It shows a structural schematic diagram of a wearable device suitable for implementing the embodiments of this application. The wearable device may include, but is not limited to, smart glasses, smartwatches, smart helmets, or any electronic device capable of performing the above functions. Figure 8 The wearable device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of this application.

[0159] like Figure 8 As shown, the wearable device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the wearable device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows wearable devices to communicate wirelessly or wiredly with other devices to exchange data. While wearable devices with various systems are shown in the figures, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0160] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0161] The wearable device provided in this application, employing the audio and video recording method described in the above embodiments, can solve the technical problem of the mismatch between the computing power resource requirements of the audio and video recording function and the actual computing power resources available for audio and video recording in the wearable device. Compared with the prior art, the beneficial effects of the wearable device provided in this application are the same as those of the audio and video recording method provided in the above embodiments, and other technical features in the wearable device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0162] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0163] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of protection of the above claims.

[0164] In addition, please refer to Figure 9 , Figure 9 This is a schematic diagram of the hardware operating environment of the terminal device involved in the audio and video recording method in this application embodiment.

[0165] This application also provides a terminal device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the steps of the audio and video recording method in the above embodiments.

[0166] The following is for reference. Figure 9 It shows a schematic diagram of the structure of a terminal device suitable for implementing the embodiments of this application. Figure 9 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0167] like Figure 9As shown, the terminal device may include a processing unit 2001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in the read-only memory 2002 or a program loaded from the storage device 2003 into the random access memory 2004. The random access memory 2004 also stores various programs and data required for the operation of the terminal device. The processing unit 2001, the read-only memory 2002, and the random access memory 2004 are interconnected via a bus 2005. An input / output interface 2006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 2006: input devices 2007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 2008 including, for example, a liquid crystal display, speaker, vibrator, etc.; storage devices 2003 including, for example, magnetic tape, hard disk, etc.; and communication devices 2009. The communication device 2009 allows the terminal device to exchange data via wireless or wired communication with other devices. Although the diagram shows terminal devices with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems may be implemented alternatively.

[0168] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 2003, or installed from read-only memory 2002. When the computer program is executed by processing device 2001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0169] The terminal device provided in this application, employing the audio and video recording method in the above embodiments, can solve the technical problem of the mismatch between the computing power resource requirements of the audio and video recording function and the actual computing power resources available for the audio and video recording function on the terminal device. Compared with the prior art, the beneficial effects of the terminal device provided in this application are the same as those of the audio and video recording method provided in the above embodiments, and other technical features in this terminal device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0170] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0171] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of protection of the above claims.

[0172] In addition, this application also provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to perform the steps of the audio and video recording method in the above embodiments.

[0173] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory, read-only memory (ROM), erasable programmable read-only memory (EPROM), optical fiber, portable compact disk ROM, optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0174] The aforementioned computer-readable storage medium may be included in a wearable device or terminal device; or it may exist independently and not assembled into a wearable device or terminal device.

[0175] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a wearable device, cause the wearable device to: respond to an input audio / video recording command, record an audio file via an MCU and a video file via an ISP; determine time synchronization information between the audio and video files; and send the audio file, video file, and target synchronization information to a terminal device, so that the terminal device, based on the target synchronization information, performs audio-video synthesis on the audio and video files to obtain an audio / video file, wherein the target synchronization information is time synchronization information, or file alignment information obtained by time-aligning the audio and video files based on the time synchronization information.

[0176] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a terminal device, cause the terminal device to: acquire audio files, video files, and target synchronization information sent by the wearable device; and, based on the target synchronization information, synthesize the audio and video files to obtain audio and video files; wherein the audio files are recorded by the wearable device in response to input audio and video recording instructions via an MCU, the video files are recorded by the wearable device in response to input audio and video recording instructions via an ISP, and the target synchronization information is time synchronization information between the audio and video files, or time alignment information obtained by aligning the audio and video files based on the time synchronization information.

[0177] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0178] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0179] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0180] The computer-readable storage medium provided in this application stores computer-readable program instructions (i.e., a computer program) for performing the steps of the above-described audio and video recording method. This solves the technical problem that the computational resource requirements of the audio and video recording function are mismatched with the actual computational resources available for audio and video recording in wearable devices. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the audio and video recording method provided in the above embodiments, and will not be repeated here.

[0181] Furthermore, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the audio and video recording method as described in the above embodiments.

[0182] The computer program product provided in this application solves the technical problem that the computing power required for audio and video recording functions is mismatched with the actual computing power available for audio and video recording functions in wearable devices. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the audio and video recording methods provided in the above embodiments, and will not be repeated here.

[0183] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. An audio and video recording method, characterized in that, The audio and video recording method is applied to a wearable device, which is communicatively connected to a terminal device. The wearable device includes a microcontroller unit (MCU) and an image signal processor (ISP). The method includes: In response to the input audio and video recording command, an audio file is recorded by the MCU, and a video file is recorded by the ISP; Determine the time synchronization information between the audio file and the video file; The audio file, the video file, and the target synchronization information are sent to the terminal device, so that the terminal device can perform audio-video synthesis on the audio file and the video file based on the target synchronization information to obtain an audio-video file. The target synchronization information is the time synchronization information, or it is file alignment information obtained by aligning the audio file and the video file in time based on the time synchronization information.

2. The audio and video recording method as described in claim 1, characterized in that, The step of determining the time synchronization information between the audio file and the video file includes: Determine the single signal transmission time between the MCU and the ISP, and determine the time offset between the audio file and the video file based on the single signal transmission time; The time offset is used as the time synchronization information between the audio file and the video file.

3. The audio and video recording method as described in claim 1, characterized in that, The step of determining the time synchronization information between the audio file and the video file includes: Obtain the first local timestamp information corresponding to the audio file and the second local timestamp information corresponding to the video file; The first local timestamp information and the second local timestamp information are used as time synchronization information between the audio file and the video file.

4. The audio and video recording method as described in claim 1, characterized in that, The step of determining the time synchronization information between the audio file and the video file includes: Obtain the MCU timestamp corresponding to the audio file, the ISP timestamp corresponding to the video file, and the timestamp mapping relationship between the MCU and the ISP; The MCU timestamp, the ISP timestamp, and the timestamp mapping relationship are used as time synchronization information between the audio file and the video file.

5. The audio and video recording method according to any one of claims 1 to 4, characterized in that, The audio and video recording instructions include a start recording instruction and a stop recording instruction. The steps of recording the target audio through the MCU and recording the target video through the ISP in response to the input audio and video recording instructions include: In response to the input start recording command, the MCU is triggered to start audio recording, and the MCU is also triggered to send a start recording signal to the ISP, wherein the start recording signal is used to trigger the ISP to start video recording; In response to the input end recording command, the MCU is triggered to send an end recording signal to the ISP to obtain the recorded video file, and the MCU is triggered to end audio recording after receiving the recording end signal returned by the ISP to obtain the recorded audio file. The end recording signal is used to trigger the ISP to end video recording and return the recording end signal to the MCU after ending video recording.

6. The audio and video recording method as described in claim 5, characterized in that, The wearable device further includes an inertial measurement unit (IMU), and the method further includes: Acquire the IMU data collected by the IMU during the recording of the video file; The IMU data is sent to the terminal device so that the terminal device performs electronic image stabilization on the audio and video files based on the IMU data.

7. An audio and video recording method, characterized in that, The audio and video recording method is applied to a terminal device, which is communicatively connected to a wearable device. The wearable device includes a microcontroller unit (MCU) and an image signal processor (ISP). The method includes: Obtain the audio files, video files, and target synchronization information sent by the wearable device; Based on the target synchronization information, the audio file and the video file are combined to obtain an audio-video file; The audio file is recorded by the wearable device in response to an input audio / video recording command through the MCU, and the video file is recorded by the wearable device in response to an input audio / video recording command through the ISP. The target synchronization information is the time synchronization information between the audio file and the video file, or it is the file alignment information obtained by aligning the audio file and the video file in time based on the time synchronization information.

8. The audio and video recording method as described in claim 7, characterized in that, The step of synthesizing the audio file and the video file based on the target synchronization information to obtain an audio-video file includes: When the target synchronization information is time synchronization information, the audio file and the video file are time-aligned based on the time synchronization information to obtain file alignment information. Then, the audio file and the video file are synthesized according to the file alignment information to obtain an audio-video file; or... When the target synchronization information is file alignment information, the audio file and the video file are combined according to the file alignment information to obtain an audio-video file.

9. The audio and video recording method as described in claim 8, characterized in that, The time synchronization information includes at least one of the following: The time offset between the audio file and the video file, wherein the time offset is determined based on the single signal transmission time between the MCU and the ISP; The first local timestamp information corresponding to the audio file and the second local timestamp information corresponding to the video file; The MCU timestamp corresponding to the audio file, the ISP timestamp corresponding to the video file, and the timestamp mapping relationship between the MCU and the ISP.

10. The audio and video recording method as described in claims 7 to 9, characterized in that, The wearable device further includes an inertial measurement unit (IMU), and the method further includes: Acquire IMU data sent by the wearable device, wherein the IMU data is data collected by the IMU during the recording of the video file; Based on the IMU data, electronic image stabilization is applied to the audio and video files.

11. A wearable device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the audio / video recording method as described in any one of claims 1 to 6.

12. A terminal device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the audio / video recording method as described in any one of claims 7 to 10.

13. A readable storage medium, characterized in that, The readable storage medium is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the audio and video recording method as described in any one of claims 1 to 6, or 7 to 10.

Images