Historical audio record display method, electronic equipment and computer readable medium

Abstract

The invention relates to the technical field of computers, in particular to a historical audio record display method, electronic equipment and a computer readable medium. The method comprises the steps that it is detected that a first triggering condition is met at the first moment, first information is displayed, and the first triggering condition is set by a user before the first moment through second electronic equipment; when it is detected that a second trigger condition is met at a second moment, second information is displayed, the first information and the second information are determined based on first audio data and second audio data collected by second electronic equipment in a recording mode, and the second trigger condition is set by the user before the second moment through the second electronic equipment. The method supports recording of related voices of daily life items or work items and the like participated by a user in a recording mode in a historical time period, displays corresponding recorded or generated first information to a specific person in a specific scene and specific time, and provides functional services of recording, memorizing, tracing and the like for the user.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method for displaying historical audio records, an electronic device and a computer-readable medium. Background Art

[0002] A time capsule (tempo kapsulo or time capsule) usually refers to putting some representative or commemorative items into a container, sealing it and burying it deep underground, and setting a time when the container can be opened in the future. It means that people put a bunch of representative items of modern inventions into a container, seal it and bury it deep underground. After decades, hundreds of years or even thousands of years, people of that era can dig it out and study it. People call this recording work placing a "time capsule" or time capsule.

[0003] With the development of computer technology, people have gradually applied some characteristics of "time capsules" (referred to as time capsule characteristics), such as the ability to store for a long time and to conduct research in the future, to terminal electronic devices. For example, people can use the terminal to take pictures, record audio and video, or enter text to record some instructions they want to give to their children or other relatives in the future, and then set the corresponding opening time. This way of writing to the future is called an electronic will. For another example, applications such as gallery applications on terminal electronic devices such as mobile phones can automatically generate past memories based on videos that have been taken or stored, and push them to users for reading.

[0004] However, the recording functions such as writing to the future or recalling the past implemented on the terminal electronic device are relatively simple. Figure 1a For example, if a user needs to recall "what did I do today?", "what did I do at location A?", "what topic did I often discuss with others last December?", etc., the above recording function cannot be realized. That is, there is currently a lack of a more systematic recording solution based on the time capsule characteristics of audio data to provide users with richer and more convenient functional services. Summary of the invention

[0005] This application provides a method for displaying historical audio records. In some application scenarios that utilize the time capsule feature, the classified and stored historical audio data is used to display the inferred information related to the user at the set time when the preset trigger condition is detected to be met. Thus, the user is provided with a rich set of functional services such as recording, keeping memories, and tracing, which is conducive to improving the user's experience of using terminal electronic devices.

[0006] In a first aspect, the present application provides a method for displaying historical audio records, which is applied to a first electronic device, the method comprising: detecting that a first trigger condition is satisfied at a first moment, and displaying first information, wherein the first information is determined based on first audio data collected by a second electronic device through recording, and the first trigger condition is set by a user through the second electronic device before the first moment; detecting that a second trigger condition is satisfied at a second moment, and displaying second information, wherein the second information is determined based on second audio data collected by the second electronic device through recording, wherein the collection time of at least part of the audio data in the first audio data is different from the collection time of at least part of the audio data in the second audio data, and the second trigger condition is set by the user through the second electronic device before the second moment.

[0007] For example, the first electronic device may be a terminal electronic device such as a mobile phone used by a user, and the second electronic device may also be a terminal electronic device such as a mobile phone used by a user, or may also be a wearable device such as a watch worn by a user. The first electronic device and the second electronic device may be the same device or different devices, and are not limited here. The first moment is the moment of displaying the first information, and the moment when the second electronic device collects the first audio data should be earlier than the first moment. Similarly, the second moment is the moment of displaying the second information, and the moment when the second electronic device collects the second audio data should be earlier than the second moment. The display method provided in the first aspect above, that is, a terminal such as a mobile phone can record the voice (i.e., audio data) related to daily life matters or work matters participated by the user by recording in different time periods of history or overlapping time periods. Furthermore, a terminal such as a mobile phone can display the first information corresponding to the historical audio data (or historical audio record) collected and stored based on the history when the preset first trigger condition is met, for users or other relevant personnel to consult.

[0008] For example, a user can record a voice or corresponding converted text (such as a letter to the future) at a certain time and set it to be displayed many years later, and the display time can be set many years later, such as a certain day 5 years, 10 years, 15 years, or 30 years later, or the user's 30th birthday or other age. In this way, when a mobile phone or other terminal detects that the display time set by the user has arrived (that is, the first trigger condition is met), it can display the user's recorded voice or a corresponding generated text-based electronic letter. For another example, a user can record N plans to be completed in 5 years by voice or text input. When the electronic device records that the "5-year" period set by the user has expired, it can push or open the N plans recorded 5 years ago for the user to verify how many have been completed, as a summary or special commemoration of the past 5 years. I will not go into details here.

[0009] In a possible implementation of the first aspect, the method further includes: detecting a first user operation indicating viewing or modifying the first information at a third moment, and displaying the first information, wherein the third moment is earlier than the first moment.

[0010] For example, when the first trigger condition such as the pre-set display time has not been met, the user wants to view or modify the first information to be displayed when the first trigger condition is met. The user can also perform some operations, including authorizing and instructing mobile phones and other terminals to display the first information and some authorization operations, to facilitate the user to use the various functions based on the time capsule characteristics provided by mobile phones and other terminals.

[0011] In a possible implementation of the first aspect, the first trigger condition and the second trigger condition are related to at least one of the user's age status and the user's operation. The user's operation may be an operation of setting a display time, an operation of instructing to display the first information, etc. The user's age status may also be used as one of the trigger conditions for the user to specify the display of the first information. For example, the user may set the display of the first information on a terminal such as a mobile phone at the age of 30 or on the birthday of a certain age.

[0012] In a possible implementation of the first aspect above, the first trigger condition includes any one of the following conditions: a condition corresponding to the trigger based on a second user operation acting on the first type of control; a condition corresponding to the trigger based on a first timing condition for viewing the first information set by the user.

[0013] In a possible implementation of the first aspect above, the first type of control includes at least one of the following: a first control that indicates the generation of first information in a target scene; a second control that sets a first trigger condition for displaying the first information in the target scene; and a third control that allows viewing of the first information after verifying user identity information in the target scene.

[0014] For example, the first type of control mentioned above can be the various functional options provided to users on the scene interface of the time capsule application running on a mobile phone or other terminal. When the user selects each functional option, the user can use the services provided under the function to record audio using the time capsule feature, extract audio content, and display some information based on the analysis results of the audio content. Rich functional options, such as "schedule review", "behavior portrait", generating "my quotations", "setting to-do", setting "my instructions", generating "meeting minutes", "emotion monitoring" and "health management", etc., can meet the user's usage needs in various scenarios (i.e., target scenarios), which is conducive to improving the user's experience of using terminal electronic devices such as mobile phones.

[0015] Among them, the functions provided by the above-mentioned "behavior portrait", "my quotations", "emotion monitoring" and "health management" function options can be another form of application of the historical audio data or historical audio records collected by the time capsule application. For example, the "behavior portrait" or "my quotations" function options can generate sentence records (i.e., quotations) related to the user's expression habits or behavior habits by analyzing the high-frequency words frequently expressed by the user extracted from the historical audio records, and then display the analyzed behavior portrait to the user, and provide some guidance and suggestions for the user to improve his or her expression habits or behavior habits. For another example, the "emotion monitoring" and "health management" function options can analyze the keywords related to emotions or moods expressed by the user in some time periods, or analyze the keywords related to physical health expressed by the user in a period of time, to form the analysis results of the user's mood state (such as emotion change curves or charts, etc.) and health state analysis results during the period, and push some warm articles or ways and methods to adjust emotions, as well as some health care suggestions for physical health, to the user.

[0016] In other embodiments, the above functions may also be implemented through another application program different from the time capsule application, which is not limited here.

[0017] In a possible implementation of the first aspect above, a fourth moment when the second electronic device collects the first audio data by recording is earlier than the first moment, and a fifth moment when the second electronic device collects the second audio data by recording is earlier than the second moment.

[0018] In a possible implementation of the first aspect above, the method also includes: detecting that a third trigger condition is satisfied at a fourth moment, instructing the second electronic device to collect the first audio data by recording, wherein the third trigger condition is set to trigger the second electronic device to start a recording function; and, detecting that the third trigger condition is satisfied at a fifth moment, instructing the second electronic device to collect the second audio data by recording.

[0019] That is, the first electronic device can instruct the second electronic device to start the recording function and record the audio data related to the user when it detects that the third trigger condition set for triggering the second electronic device to start the recording function is met. It can be understood that if the first electronic device and the second electronic device are the same device, the first electronic device can trigger the recording function to record audio data when it detects that the third trigger condition is met.

[0020] In a possible implementation of the first aspect above, the third trigger condition includes any one of the following conditions: a condition corresponding to the trigger based on a third user operation acting on the second type of control to indicate turning on the recording function; a condition corresponding to the trigger based on the second timing condition for turning on the recording function set by the user; a condition corresponding to the trigger based on the location condition for turning on the recording function set by the user; and a condition corresponding to the trigger based on the wearing condition for turning on the recording function set by the user.

[0021] For example, the second type of control may be a recording control displayed on a mobile phone or other terminal, and the recording control may be provided on the interface of a time capsule application running on a mobile phone or other terminal, or provided on an interface of a control center, etc., without limitation. The location condition may be, for example, a trigger location set by a user to trigger the recording function.

[0022] In a possible implementation of the first aspect above, the first information and the second information include any one of the following: information that is commemorative for the user; information that serves as a reminder for the user; information that serves as a traceability for the user; and information that serves as a record for the user.

[0023] For example, the above-mentioned information that is of commemorative significance to the user may be information displayed corresponding to functions such as setting "My Instructions", generating "My Quotes", and reviewing schedules provided by the time capsule application running on mobile phones and other terminals. The above-mentioned information that has a reminder effect on the user may be information displayed corresponding to functions such as setting to-do items, setting "My Instructions", and generating "My Quotes" provided by the time capsule application running on mobile phones and other terminals. The above-mentioned information that has a traceability effect on the user may be information displayed corresponding to functions such as reviewing schedules, behavioral profiling, emotional monitoring, and health management provided by the time capsule application running on mobile phones and other terminals. The above-mentioned information that has a recording effect on the user may be information displayed corresponding to functions such as generating meeting minutes provided by the time capsule application running on mobile phones and other terminals. No limitation or elaboration is made here.

[0024] In a possible implementation of the first aspect above, the first electronic device and the second electronic device include any one of the following devices: a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a smart watch, a smart bracelet, a smart headset, a smart TV, and a smart speaker.

[0025] In a second aspect, the present application provides an electronic device comprising: one or more processors; one or more memories; one or more memories storing one or more programs, which, when the one or more programs are executed by the one or more processors, enables the electronic device to execute the method for displaying historical audio records provided in the above-mentioned first aspect and various possible implementations of the first aspect.

[0026] In a third aspect, the present application provides a computer-readable medium having instructions stored thereon, which, when executed on a computer, causes the computer to execute the method for displaying historical audio records provided in the first aspect and various possible implementations of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1a Shown is a schematic diagram of a recording scenario provided in an embodiment of the present application.

[0028] Figure 1b Shown is a schematic diagram of a multi-device scenario related to the implementation of a method for displaying historical audio records provided in an embodiment of the present application.

[0029] Figure 2a Shown is a schematic diagram of a scene interface provided by a time capsule application provided in an embodiment of the present application.

[0030] Figure 2b Shown is a schematic diagram of a schedule review interface provided by a time capsule application provided in an embodiment of the present application.

[0031] Figure 2c Shown is a schematic diagram of a recording interface provided by a time capsule application provided in an embodiment of the present application.

[0032] Figure 2d Shown is a schematic diagram of an audio details interface provided by a time capsule application provided in an embodiment of the present application.

[0033] Figure 2e Shown is a schematic diagram of a device interface provided by a time capsule application provided in an embodiment of the present application.

[0034] Figure 2f Shown is a schematic diagram of a classification interface provided by a time capsule application provided in an embodiment of the present application.

[0035] Figure 2g Shown is a schematic diagram of an audio list interface provided by a time capsule application provided in an embodiment of the present application.

[0036] Figure 3 The figure is a schematic block diagram of the software system structure of a mobile phone provided in an embodiment of the present application.

[0037] Figure 4 The figure is a schematic diagram of the implementation process of a method for displaying historical audio records provided in an embodiment of the present application.

[0038] Figure 5 Shown is a schematic diagram of a "control center" interface displayed on a mobile phone provided in an embodiment of the present application.

[0039] Figure 6 The figure shows a schematic diagram of the implementation process of a method for displaying historical audio records provided in Example 1 of the present application in a functional scenario corresponding to schedule review.

[0040] Figure 7a Shown is a schematic diagram of a schedule review interface displayed on a mobile phone provided in Example 1 of the present application.

[0041] Figure 7b Shown is a schematic diagram of another schedule review interface displayed on a mobile phone provided in Example 1 of the present application.

[0042] Figure 7c Shown is a schematic diagram of an audio details interface displayed on a mobile phone provided in Example 1 of the present application.

[0043] Figure 8 The figure shows a schematic diagram of the implementation process of a method for displaying historical audio records provided in Example 2 of the present application in a functional application scenario corresponding to setting to-do items.

[0044] Figure 9a The figure shows a schematic diagram of the implementation process of a method for displaying historical audio records provided in Example 3 of the present application in a functional scenario corresponding to a behavior portrait.

[0045] Figure 9b Shown is a schematic diagram of a catchphrase interface displayed on a mobile phone provided in Example 3 of the present application.

[0046] Fig.10a The figure shows a schematic diagram of the implementation process of a method for displaying historical audio records provided in Example 4 of the present application in a functional scenario of emotion monitoring or health management.

[0047] Fig.10b Shown is a schematic diagram of an emotion management interface displayed on a mobile phone provided in Example 4 of the present application.

[0048] Fig.11 The figure shows a schematic diagram of the implementation process of a method for displaying historical audio records provided in Example 5 of the present application in a functional scenario corresponding to meeting minutes.

[0049] Fig.12 Shown is a schematic diagram of the hardware structure of a mobile phone provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0051] As mentioned above, the recording functions currently implemented on terminal electronic devices using the time capsule feature are relatively simple, lacking a more systematic recording solution based on the time capsule characteristics of audio, video and other data to provide users with richer and more convenient functional services.

[0052] In order to solve the above problems, the present application provides a method for displaying historical audio records, which is applied to mobile phones, tablet computers, desktops, laptops, handheld computers, augmented reality (AR) / virtual reality (VR) devices, wearable devices such as smart watches, smart TVs, smart speakers, servers, televisions with one or more processors, or other terminal electronic devices that can access the Internet.

[0053] Specifically, the method uses classified and stored historical audio data to display the estimated information related to the user at a set time in some application scenarios that utilize the time capsule feature when the preset trigger condition is detected to be met. The specific information can be recorded in the form of a time capsule and displayed to the user at a specific time (such as the above-mentioned set time). The specific information includes specific content recorded or expressed by the user by recording audio, such as describing meaningful things done by oneself in a specific time period, a letter to the future, or expressing one's entrustment or expectation to people who can view the estimated information in the future. Correspondingly, the above-mentioned trigger condition can be, for example, a timed trigger condition set by the user, and the above-mentioned set time can be, for example, many years later set by the user, such as 5 years, 10 years, 15 years, 30 years later, etc. The above-mentioned specific information can also include information related to the user generated by extracting historical audio data recorded by the user, such as user quotations corresponding to the historical audio data extracted in a certain time period, or corresponding predicted user mood state or physical health state and other related information. The specific information can be used as a user or other relevant personnel to display user quotations sealed at a specific time point in the past in a specific scenario in the future, or predicted user mood state or physical health state and other related information to achieve the recording purpose. Correspondingly, the above-mentioned preset triggering conditions may be, for example, related conditions such as changes in the user's age status, marital status, and physical health status.

[0054] For example, a user can record a voice or corresponding converted text (such as a letter to the future) at a certain time and set it to be displayed many years later, and can set the display time many years later (i.e. the time set above), such as a certain day 5, 10, 15, or 30 years later, or the user's 30th birthday or other age. In this way, when a mobile phone or other terminal detects that the display time set by the user has arrived, it can display the user's recorded voice or a corresponding generated text-based electronic letter. For example, a user can record N plans to be completed in 5 years by voice or text input. When the electronic device records that the "5-year" period set by the user has expired, it can push or open the N plans recorded 5 years ago for the user to verify how many have been completed, as a summary or special commemoration of the past 5 years.

[0055] For example, refer to Figure 2a The "My Instructions" function shown in the figure allows users to set the conditions for opening the corresponding recorded audio data at a specific time in the future, and through the control for triggering recording set in this function, jump to the recording interface to record the audio data to be sealed. Then, the audio data that needs to be sealed in the time capsule is recorded. Figure 2a The "user quotations" function can extract content related to different life stages such as age status and marital status based on the text content extracted from the audio data recorded by the user in certain historical time periods, generate quotations that the user may need to check by the user himself or the user's relatives at a specific age or marital status, remind the user to set the opening time, and seal the content as a commemorative time capsule record.

[0056] For another example, based on the display method provided in this application, the electronic device can also provide Figure 2a The "Schedule Review" function shown in FIG. Under this function, the electronic device can review the user's previous schedule based on the text content converted from the audio data. In addition, refer to Figure 2a As shown in the "emotion monitoring" and "health management" functions, the electronic device can also provide users with emotion monitoring, health management and other functional services based on the corresponding audio features extracted from the audio data that can reflect the tone of voice and physical condition, such as "sighs", "coughs", "hoarseness" and other audio features with special changing rules. Figure 2a As shown in the “meeting minutes” function, the electronic device can also automatically generate meeting minutes based on the above text content.

[0057] For example, refer to Figure 2aThe "Set To-Do" function shown in the figure can automatically set reminders for users based on the future plans and other contents in the text content converted from the audio data, and push them to the user for confirmation. At this time, the electronic device can become the user's personal secretary, improving the user's experience. In some embodiments, the electronic device can also provide "behavior portraits", "meeting minutes" and other functional services based on the display method of historical audio records provided by this application. In some embodiments, reference Figure 2a The classification options shown can mark the corresponding person, such as the user or other person who has communicated with the user, based on the voiceprint features in the audio features corresponding to the audio data, and then extract the audio data records participated by the person to help the user or other person who has communicated recall what they said in the past. The details will be described in detail below in conjunction with the corresponding figures, and will not be repeated here.

[0058] It can be understood that the classified storage of historical audio data can be, for example, a method in which the electronic device extracts audio features and converts the audio data collected in the user's daily life and work process into text before storing it, and adds corresponding keyword tags to the recorded audio data for classification and storage. In this way, based on the classified stored audio data, the electronic device can provide users with a variety of recording function services based on the above audio data, such as the functions provided to users in the above-mentioned various application scenarios using the time capsule characteristics. For details, please refer to Figure 2a As shown, the electronic device provides a variety of recording function services to the user and can display the corresponding interface through the running application or system service to provide the user with the corresponding function service operation entrance, refer to Figure 2a The scene interface shown shows option entries corresponding to each scene function. When the user clicks the option entry corresponding to each scene function, the electronic device can display the operation interface that provides corresponding functional services under the corresponding scene function, and can provide the user with corresponding functional services based on the user's operation on the corresponding operation interface.

[0059] It is understandable that the above audio data collection process can be automatically collected by electronic devices such as mobile phones when the set recording conditions are triggered. In addition, the electronic device can synchronously record the location at the time of collecting the corresponding audio data, such as positioning information. The above recording conditions can be set by the user or preset in the operating system of the electronic device. For example, the user can complete the setting through the recording interface displayed by the mobile phone running the time capsule application. For details, please refer to the following Figure 2bThe above-mentioned recording conditions can be timing trigger conditions, conditions that trigger the start of recording based on user operations, conditions that trigger the start of recording based on wearing detection results, etc., and are not limited here. For example, the electronic device can determine that the recording condition for starting recording is detected when it detects that the user clicks on the recording control, the collection time set by the user arrives, the user arrives at the preset trigger location, or detects the user wearing a wearable device such as a watch. The above-mentioned recording controls, for example, can be dialogue controls or input controls that can call the input method soft keyboard, etc., and are not limited here.

[0060] It can be understood that before the electronic device classifies and stores the collected audio data, it can extract multi-dimensional feature information from the collected audio data, including but not limited to voiceprint feature information, positioning information, and environmental feature information extracted based on the corresponding converted text content and the corresponding audio features. In addition, in the process of adding the above-mentioned keyword tags to the audio data for classified storage, the read address corresponding to each audio data can also be added to the corresponding classification information or classification record table, so that when the electronic device responds to the user selecting the corresponding scene entrance to use the corresponding functional service, the corresponding audio data and the corresponding converted text record are called, etc., to provide the user with the content displayed in support of the corresponding functional service.

[0061] It can be understood that in the embodiment of the present application, the above-mentioned application that can provide users with multiple recording function services based on the above-mentioned audio data can be, for example, a time capsule application or other system application or third-party application that integrates the above-mentioned multiple recording function services, such as a recording application.

[0062] It can be understood that the above-mentioned electronic devices for collecting audio data can be electronic devices such as mobile phones and tablets carried by users, or wearable devices such as smart watches, bracelets, smart glasses worn by users, or smart speakers, smart voice assistants, smart TVs and other electronic devices with sound pickup functions arranged in the user’s home space or other environments, without limitation here.

[0063] In addition, while running the time capsule application to provide the above-mentioned functional services, mobile phones and other terminals can also display recommended content associated with the corresponding functions to users. For example, when the terminal analyzes the user's recent mood by providing an emotion monitoring function, or provides a health management function to analyze the user's mental health status during a certain period of time, it can analyze the user's emotional cycle based on the audio data collected during the corresponding time period and predict the user's next emotional trend, etc. Correspondingly, the terminal can display some emotion regulation methods or suggestions for users to review. In this way, mobile phones and other terminals can use the time capsule characteristics of audio data based on the display method of historical audio records provided by this application to record users' daily related and life trajectories, and provide users with convenient recording solutions such as queries and reminders.

[0064] It is understood that the audio data based on which the method for displaying historical audio records provided by the present application is implemented can be stored in the local memory of electronic devices such as mobile phones after collection, or can be uploaded to a server through wireless communication for classified storage and management. Figure 1b As shown, the user can also record audio data through the watch 300a worn by the user, or record audio data through the speaker 300b. The audio data collected by the watch 300a or the speaker 300b can be sent to the mobile phone 100 for storage based on the communication connection between the mobile phone 100, or can be uploaded to the server 200 for storage. There is no limitation here. At the same time, the location information of the electronic device collecting audio corresponding to the time of collection can also be stored together with the collected audio data. Among them, the mobile phone 100 can be installed with an application with time capsule characteristics, such as Figure 1b The Time Capsule app shown.

[0065] As an example, refer to Figure 2a As shown, the mobile phone 100 running the time capsule application can display the scene interface 210. The scene interface 210 can provide multiple application scenarios. The user clicks on the corresponding scene option to use the functional services based on the time capsule characteristics in the corresponding scenario, such as "schedule review", "behavior portrait", generating "my quotations", "setting to-do", setting "my instructions", generating "meeting minutes", "emotion monitoring" and "health management". For example, the user can click Figure 2a The schedule review 211 shown, the mobile phone 100 can respond to this operation by displaying Figure 2b A schedule review interface 220 is shown.

[0066] Continue to refer Figure 2bAs shown, the user can click the dialogue control 221 on the schedule review interface 220 and issue a voice instruction to perform a retrospective search to view the historical schedule records. For example, the user can say "What did I do today?" after clicking the dialogue control 221, and the mobile phone 100 can respond to the voice instruction by displaying the text content corresponding to the audio data recorded today. The text content can be, for example, Figure 2b The content related to "December 24, 2022" shown includes that the user may have gone to "Location A" today to do two things, namely, "discussed technical solution Y001 with X001" and "discussed technical solution Y002 with X002", and may have also gone to "Location B" to do one thing, namely, "had lunch with X001 and X002". Further, the user clicks Figure 2b The text content shown can also cause the mobile phone 100 to further jump to the associated audio data display interface, which can refer to the following Figure 2d The following descriptions are not given here. Figure 2b As shown, the text content displayed by the mobile phone 100 can be presented in the form of a mind map, a tree diagram, a fishbone diagram, a timeline diagram, etc., which is not limited here.

[0067] Continue to refer to the above Figure 2a As shown, the interface displayed by the mobile phone 100 when running the time capsule application program may include, in addition to the above-mentioned scene interface 210, a recording interface (corresponding to Figure 2a Recording options 212 shown), device interface (corresponding to Figure 2a Device options 213 shown) and classification interface (corresponding to Figure 2a The classification options 214 shown). Correspondingly, the user clicks Figure 2a The recording option 212, device option 213, and classification option 214 shown can open the corresponding interface.

[0068] As an example, the recording interface displayed by the mobile phone 100 can refer to Figure 2c The recording interface 230 is shown. Figure 2cAs shown, the recording interface 230 may display a recording switch 231, an audio record 232, and a recording setting 233. Among them, the user can trigger the start or end of recording by clicking the recording switch 231. The naming method of each audio record in the audio record 232 is named and saved, for example, "20230106_153411", "20230105_153410", "20230104_153409", "20230102_153408", "20230101_153407", etc. In other embodiments, the recorded audio data can also be saved using other naming methods, which are not limited here. The user can also click the recording setting 233 to set the trigger condition for triggering the start of the recording switch, which will not be repeated here.

[0069] Further, when the user clicks on an audio record in the audio record 232, for example, when the user clicks on the audio record named "20230106_153411", the mobile phone 100 can display a corresponding Figure 2d The audio details interface 240 is shown. Figure 2d As shown, the name 241 of the corresponding audio, such as "20230106_153411", can be displayed on the audio details interface 240. The user can click the audio playback control 242 on the audio details interface 240 to play the audio. The audio details interface 240 can also display locations 243 (including cities, districts, counties, etc.) determined based on GPS or operator data, subdivided locations 244 (including restaurants, museums, etc.) determined based on environmental features extracted from the audio data, people 245 marked based on the voiceprint features extracted from the audio, and topics 246 discussed by relevant people analyzed based on the text content converted from the audio, etc. Among them, the above-mentioned places 243 include, for example, "BJ", "SH", etc., and further subdivided places 244 include, for example, "restaurant", "museum", etc.; the above-mentioned characters 245 include, for example, "character A", "character B", etc., and the topics 246 discussed by the above-mentioned characters include, for example, "topic A", "topic B", etc., where topic A can be related to "what to eat", and topic B can be related to "what to see", which will not be elaborated here.

[0070] As an example, the device interface displayed by the mobile phone 100 can refer to Figure 2e The device interface 250 shown. Figure 2e As shown, the user can click the "add device" control 251 or control 252 on the device interface 250 to add a device that can cooperate with the recording, such as the watch X-01, the speaker Y-01, etc. Figure 2eAs shown, the added device information can be displayed in the added device list 253 of the device interface 250. It can be understood that the mobile phone 100 and the added devices can form a distributed recording system to better collect the daily conversations and audio data in the scene in which the user participates. When the user does not need to use other devices to record, he can also Figure 2e The delete control 254 on the device interface 250 shown deletes the corresponding "speaker Y-01" etc. This is not limited here.

[0071] As an example, the classification interface displayed by the mobile phone 100 can refer to Figure 2f The classification interface 260 shown. Figure 2f As shown, the classification interface 260 can display the classification results corresponding to the features such as text extracted from the audio data, the height and change of the sound frequency, etc. The mobile phone 100 can identify the person recorded in the audio data, the tone of the person's speech, whether there is environmental characteristic audio related to the surrounding environment, etc. based on the above extracted features, and then combine the location information recorded when collecting the audio data, etc., to further classify the recorded or collected audio data. Figure 2f As shown, for example, if the user clicks on the character option 261 on the classification interface 260, the mobile phone 100 can display audio data associated with characters such as "character A", "character B", etc. classified based on voiceprint features. For another example, if the user clicks on the place option 262 on the classification interface 260, the mobile phone 100 can display places such as "BJ", "SH", etc. classified based on location features, as well as "restaurant", "museum", etc. further subdivided based on environmental features ( Figure 2f For another example, if the user clicks on the topic option 263 on the classification interface 260, the mobile phone 100 can display the topics discussed by the user or other characters based on the text content extraction, such as "what to eat", "what to watch", etc. ( Figure 2f The audio data associated with the device (not shown) etc. For details, please refer to the detailed description below, which will not be repeated here.

[0072] Furthermore, if the user clicks "Character A" on the classification interface 260, the mobile phone 100 can further display Figure 2g The audio list interface 270 associated with the character A shown in the figure. The interface 270 may display one or more audio data records in which the "character A" participates, such as "20230106_153411", "20230104_153409", "20230102_153408", etc. Figure 2g Clicking the audio record "20230106_153411" on the interface 270 shown in the figure, the mobile phone 100 can also display the above Figure 2d The audio details interface 240 is shown.

[0073] In other embodiments, the recording interface may also be different from the above Figure 2c Other interface styles shown, the above device interface can also be different from the above Figure 2e Other interface styles shown, the above classification interface can also be different from the above Figure 2f Other interface styles shown are not limited here.

[0074] It is understandable that electronic devices can use the time capsule feature through time capsule applications and the like, and use the recorded audio data to correspond to the recorded content, which can greatly facilitate users to trace or assist users in memorizing. In some scenarios, the above-mentioned method for displaying historical audio records based on the time capsule feature provided by this application, in addition to providing the above-mentioned Figure 2b In addition to the schedule review function service exemplified, it can also provide users with a variety of other functional services, such as the above-mentioned "behavior profiling", "setting to-do", "meeting minutes", "emotion monitoring" and "health management" and other functional services. Among them, setting to-do can also be described as a private secretary in other embodiments, which is used to set to-do reminders. The above-mentioned "meeting minutes" can realize the automatic generation of meeting minutes based on the recorded meeting-related audio data. The meeting minutes can also be presented in text forms such as mind maps, tree diagrams, fishbone diagrams, etc., which are not limited here. The above-mentioned "emotion monitoring" can extract audio features based on the audio data collected historically to monitor the user's mood changes or emotional cycles, etc. The above-mentioned "health management" can include functions such as physical health management and mental health management, among which mental health management can, for example, provide mental health analysis and provide psychological counseling opinions and other functions. The details will be described in detail below in conjunction with the embodiments, and will not be repeated here.

[0075] The technical effects of the present application are not limited to the above-mentioned effects, and those skilled in the art can also clearly understand other effects not mentioned through the description of the following embodiments.

[0076] It can be understood that the electronic devices applicable to the method for displaying historical audio records provided in the embodiments of the present application may include but are not limited to mobile phones, tablet computers, desktops, laptops, handheld computers, netbooks, as well as augmented reality (AR) / virtual reality (VR) devices, smart TVs, smart watches and other wearable devices, servers, mobile email devices, car devices, portable game consoles, portable music players, reader devices, televisions with one or more processors, or other electronic devices that can access the Internet.

[0077] Exemplarily, the wearable devices in the embodiments of the present application can be wired headphones, wireless headphones (for example, TWS Bluetooth headphones, neck-hanging Bluetooth headphones, head-mounted Bluetooth headphones), smart watches, smart bracelets, smart glasses, smart anklets, smart rings, smart necklaces, AR devices, VR devices, etc. The present application does not impose any special restrictions on the specific form of the wearable device.

[0078] Next, taking mobile phone 100 as an example, the specific implementation process of the method for displaying historical audio records provided by the present application is described in detail.

[0079] For example, Figure 3 According to an embodiment of the present application, a schematic block diagram of the software system structure of a mobile phone is shown.

[0080] like Figure 3 As shown, the software system of the mobile phone 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. TM Taking the system as an example, the software system structure of the mobile phone 100 is illustrated.

[0081] The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, Android TM The system is divided into four layers, from top to bottom: application layer, application framework layer, Android TM Runtime (Android TM runtime) and system libraries, as well as the kernel layer.

[0082] Among them, the application layer may include a series of application packages. The application package may include time capsule, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications. In an embodiment of the present application, the implementation process of the above-mentioned method for displaying historical audio records can be realized by the above-mentioned time capsule application, and the various functional services corresponding to the time capsule characteristics can be integrated in the time capsule application. In other embodiments, the recording and audio data processing processes corresponding to the time capsule application and the various functional services corresponding to the time capsule characteristics can also be integrated in some system applications or system services installed on the mobile phone 100. Among them, the system application can be, for example, a recording application, a setting application, etc., and the system service can be, for example, a time management service or a media controller provided by the following application framework layer. There is no limitation here.

[0083] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0084] The application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and a time management service.

[0085] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0086] Content providers are used to store and retrieve data and make it accessible to applications. This data can include video, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0087] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0088] The phone manager is used to provide communication functions of the mobile phone 100, such as management of call status (including answering, hanging up, etc.).

[0089] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0090] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0091] The time management service provides system-level audio processing capabilities, performs feature extraction and classification on the collected audio data, and forms callable historical data for implementing the time capsule function. It then provides functional services based on the time capsule characteristics for the time capsule application or recording application, setting application, etc. at the application layer. For details, please refer to the relevant description below, which will not be repeated here.

[0092] Android TM Runtime includes core libraries and virtual machines. Android runtime is responsible for Android TM Scheduling and management of the system.

[0093] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android TM The core library.

[0094] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0095] The system library may include multiple functional modules, such as surface manager, media libraries, 3D graphics processing library (such as openGL ES), 2D graphics engine (such as SGL), etc.

[0096] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0097] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as the formation of static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. In the embodiment of the present application, the media library can support the recording (or collection) and processing of audio data using the time capsule feature, as well as audio playback and other functions required in the process of implementing some functional services based on the time capsule feature, which are not limited here.

[0098] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0099] A 2D graphics engine is a drawing engine for 2D drawings.

[0100] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0101] It should also be stated that the steps in the methods and processes in the embodiments of the present application are numbered for ease of reference, rather than to limit the order of precedence. If there is an order between the steps, the text description shall prevail.

[0102] Figure 4According to an embodiment of the present application, a schematic diagram of an implementation flow of a method for displaying historical audio records is shown.

[0103] like Figure 4 As shown, the implementation process may include the following steps:

[0104] 410: The mobile phone 100 obtains user authorization for collecting audio data.

[0105] Exemplarily, when the mobile phone 100 runs the Time Capsule application for the first time after installing it, the mobile phone 100 may display an authorization interface. The authorization interface may display, for example, a privacy agreement and an authorization confirmation control to remind the user to confirm whether the device is allowed to record while the application is running, and to remind the user to confirm whether the application is allowed to run in the background for a long time. The privacy agreement may involve the collection of voice, GPS and other data, as well as the processing and application of the collected voice information (i.e., audio data), GPS information, etc., for the user to review. The style of the above-mentioned specific authorization interface can refer to the authorization interface style of some current applications, and is not limited here.

[0106] In other embodiments, the electronic device such as the mobile phone 100 may also run a system service with a time capsule function by default, and the shortcut function area of ​​the mobile phone 100 may display a corresponding function control, which may be a system application that provides a time capsule function service independently, or may be integrated into an installed system application. Figure 5 As shown, for example, a user slides down on the screen of the mobile phone 100 to display a "time capsule" control 511 or a "recorder" control 512 with a time capsule function. When the mobile phone 100 responds to the user clicking the "time capsule" control 511 or the "recorder" control 512, and uses the system application to trigger the start of a system service with a time capsule function for the first time, the mobile phone 100 may also display a corresponding authorization interface. This is not limited here.

[0107] It is understood that the device used to collect audio data can also be an electronic device such as a watch, earphones, and speakers. The user can add one or more electronic devices authorized to collect audio data on the device authorization interface provided by the running Time Capsule application. For the specific operation interface of adding audio collection devices, please refer to the above Figure 2d As shown, the operation process can refer to the above Figure 2d The relevant description will not be repeated here.

[0108] It is understandable that the electronic device added by the user can establish a communication connection with the mobile phone 100, and the communication connection can be established based on Wi-Fi, Bluetooth, or a distributed soft bus. In other embodiments, the user can also continue to confirm the authorization operation on the added electronic devices such as watches and speakers, for example, the user presses the corresponding control to perform the authorization operation according to the voice prompt of the electronic device, etc., which is not limited here.

[0109] In other embodiments, the mobile phone 100 implements the method for displaying historical audio records provided by the present application, and can also be performed after obtaining the user's default authorization. For example, the mobile phone 100 can perform the following steps 420 to 440 to implement the method for displaying historical audio records provided by the present application.

[0110] 420: The mobile phone 100 detects that the recording switch is triggered and turns on, and collects audio data.

[0111] For example, the recording switch may be a switch triggered by a control such as a recording control, or may be based on a preset time condition, location condition, scene condition, or wearing trigger condition. Figure 2a The recording control 211 shown triggers the recording switch to enable the mobile phone 100 to collect audio data. For another example, the mobile phone 100 can also automatically trigger the recording switch to collect audio data when it detects that the preset time to turn on the recording switch has arrived.

[0112] For another example, when the mobile phone 100 detects that the currently collected location information matches the preset location information for triggering the recording switch, it can automatically trigger the recording switch to collect audio data. Alternatively, the mobile phone 100 can also trigger the recording switch to collect audio data when it detects that the current user is in a preset scene such as a museum or restaurant based on the scene recognition function and the audio and location information collected by the scene recognition function. This is not limited here.

[0113] For another example, in a scenario where the mobile phone 100 collects audio data through a watch worn by a user, the watch can detect whether the user is wearing a watch or whether there is an action of wearing a watch based on preset wearing detection sensors, such as infrared light sensors and inertial sensors (IMU) such as acceleration sensors and gyroscope sensors. If the watch detects the action of the user wearing the watch, it can trigger the recording switch to turn on to collect audio data, that is, start recording voice. Correspondingly, if the watch does not detect that the user is wearing the watch for a long time (for example, more than a preset time), or the watch detects the action of the user taking off the watch, it can trigger the recording switch to turn off to stop recording.

[0114] It is understood that the user can set the time conditions, location conditions, scene conditions and wearing trigger conditions for triggering automatic voice data collection through the time capsule application running on the mobile phone 100, or the system service with the time capsule function, so as to trigger the recording switch to turn on and automatically collect audio data when the corresponding trigger conditions are met. Figure 2c In the recording interface 230 shown, for example, the user can click the recording setting 233 to set the above-mentioned various trigger conditions.

[0115] In some embodiments, the mobile phone 100 may also detect whether the condition set by the user not to record audio data is met, for example, the recording switch may be turned off when the user enters a preset private place, such as a conference room, home, etc. Correspondingly, the mobile phone 100 may automatically trigger the recording switch to be turned on when the user leaves the preset private place.

[0116] In other embodiments, the user can also set the recording switch to be turned on in a private space through a recording control or a preset shortcut gesture. For example, in a conference room, the user can trigger the recording switch to be turned on and collect audio data by clicking the recording switch displayed on the mobile phone 100, or clicking the volume button or the power button of the mobile phone 100 according to a preset number of times, or tapping the screen of the mobile phone 100 according to a preset number of times, etc., without limitation. The user can also trigger the mobile phone 100 to turn on the recording switch by saying a preset voice wake-up word, such as "open time capsule", etc., and when the recording needs to be ended, the recording can also be ended by voice command, which is also not limited here.

[0117] It can be understood that the way to trigger the recording switch to start audio data collection can be a variety of ways and a combination of a variety of ways, not limited to the various ways in the above examples.

[0118] It is understood that when there are multiple electronic devices authorized to collect audio, the mobile phone 100 can also select the device that is currently most suitable for collecting audio data from multiple electronic devices based on the wearing status of each electronic device, the distance between the user (affecting the intensity of the collected audio data), the corresponding signal-to-noise ratio when collecting audio data, etc. The corresponding collected audio data can be transferred to the mobile phone 100 for storage or uploaded to the corresponding server for storage based on the communication connection with the mobile phone 100, such as Wi-Fi, Bluetooth or distributed soft bus.

[0119] It is understood that when the recording switch provided by the time capsule application or service is activated on the mobile phone 100 to collect audio data, the mobile phone 100 can make corresponding prompts, such as displaying a pop-up notification to prompt the start, broadcasting a voice prompt such as "You have entered the time capsule" or "You have exited the time capsule", or displaying the status indicator light constantly, with a breathing effect, flashing, and color changing effects. In other embodiments, the mobile phone 100 can also remind the user to enter or exit the state of recording audio data (i.e., recording) in the time capsule through reminders such as vibration prompts, which are not limited here.

[0120] In addition, it can be understood that when the sound of the user's position changes, the electronic device for collecting audio data can also switch from the current device to another electronic device, such as switching from the mobile phone 100 to the watch to collect audio data. When switching, the watch or mobile phone 100 can also remind the user that the recording device has been switched through the above reminder method or other reasonable methods, which is not limited here.

[0121] 430: The mobile phone 100 performs feature extraction and other processing on the collected audio data, and classifies and stores the data.

[0122] Exemplarily, the mobile phone 100 may extract audio features from the collected audio data. The extracted audio features may include, but are not limited to, feature information in the following multiple dimensions:

[0123] Voiceprint features are used to mark people. For example, the corresponding audio data can be marked according to different voiceprint features. Figure 2b Shown are character A, character B, etc.

[0124] Environmental features are used to match the location or environment of the user, so as to further determine the subdivided location in combination with positioning information. A pre-trained scene recognition model can be set in the mobile phone 100 to identify the corresponding scene based on the audio features of specific frequencies and specific timbres related to the scene in the collected audio data.

[0125] Based on the collected audio data, audio features related to health can also be extracted, such as "coughing", "yawning", "nasal sound when having a cold", etc., as a basis for judging the health status of the user or other people. In this way, the mobile phone 100 can also recommend medical guidance, suitable diet and activities, etc. according to the health status of the user. For details, please refer to the relevant description below, which will not be repeated here.

[0126] In addition, the collected audio data can be converted to corresponding text content, which can be used to further extract the paragraph gist, keywords, and topics of discussion of the content recorded in the corresponding audio data. The mobile phone 100 can also be provided with a pre-trained semantic recognition model to recognize the text recorded in the audio data and convert the audio data into corresponding text content or text records, thereby extracting the paragraph gist, keywords, and topics of discussion.

[0127] Based on the text content obtained by the above conversion, the mobile phone 100 can further identify some of the interjections, such as "ah", "haha", "hehe", etc., and then based on these interjections, recognize the emotions of the user recorded in the corresponding voice data and the emotions of other people around the user.

[0128] In addition, based on the converted text content, the mobile phone 100 can further identify the text that can trigger preset to-do items, schedule reminders, anniversaries, birthdays, etc., and then trigger the corresponding reminders, which is conducive to improving the user experience. For details, please refer to the relevant description below, which will not be repeated here.

[0129] The audio data collected above may be collected by the mobile phone 100 itself, or may be collected by an electronic device such as a watch authorized by the mobile phone 100. The audio data may be original collected data or compressed audio data, etc., which is not limited here.

[0130] It can be understood that before extracting features from the audio data, the mobile phone 100 can also perform pre-processing such as noise reduction and filtering on the collected audio data to improve the accuracy of feature extraction. After completing the feature extraction, the mobile phone 100 can add classification labels corresponding to various types of feature information to the corresponding audio data, such as adding location labels such as "BJ" and "SH", or character labels such as "Character A" and "Character B", or scene labels such as "restaurant" and "museum", etc., and can classify and store the corresponding audio data based on the added classification labels. The classification storage method can, for example, record the storage address of the audio data in the memory of the mobile phone 100 into the data table corresponding to each classification for easy access. The results of the above-mentioned classification storage can also be displayed on the mobile phone 100 accordingly. Refer to the above Figure 2e The classification interface 250 shown can display the classification results corresponding to the above classification labels respectively.

[0131] In other embodiments, the mobile phone 100 may classify and store the collected audio data in other reasonable ways, which are not limited here.

[0132] 440: The mobile phone 100 detects user operations in different scenarios or the conditions for triggering display set by the user are met, and displays corresponding audio data or content extracted based on the audio data.

[0133] Exemplarily, based on the extracted audio features and the classified and stored audio data, the mobile phone 100 can provide the user with some functional services based on the time capsule characteristics, such as schedule review (or retrospective retrieval) function, setting my instructions (recording voice to be opened in the future), setting to-do reminders (or private secretary) function, emotion monitoring function, health management function, meeting record (such as generating meeting minutes) function, and behavior portrait (such as catchphrases or common words in a certain period of time) recognition, generating my quotations (forming text or voice records to be opened in the future), etc. These functional services need to call the corresponding audio data and some audio features extracted based on the audio data, so when the mobile phone 100 detects that the user operates on the corresponding interface or wakes up the above functional services by voice, etc., or when the mobile phone 100 detects that the trigger conditions set by the user to display specific information in a specific scene or at a specific time are met, the mobile phone 100 can display or broadcast the function indicated by the user to be awakened, or the content matching the information set by the user to be displayed.

[0134] For the specific process of calling the recorded audio data in different scenarios to provide corresponding display functions, please refer to the relevant descriptions in the following embodiments, which will not be repeated here.

[0135] Based on the above Figure 4 As shown in the implementation process, the method for displaying historical audio records provided by this application can give full play to the time capsule characteristics, use audio data to record the user's life, work, etc., and provide users with a variety of convenient functional services based on the time capsule characteristics, which is conducive to improving the user's experience of using terminals such as mobile phones, watches, speakers, and time capsule applications.

[0136] The following describes in detail the specific implementation process of the method for displaying historical audio records provided by the present application in different application scenarios in combination with different embodiments.

[0137] Example 1

[0138] The embodiment of the present application provides a specific implementation process of implementing the method for displaying historical audio records provided by the present application in a retrospective scenario of schedule review, such as retrieving daily conversations.

[0139] Figure 6 According to an embodiment of the present application, a schematic diagram of the implementation process of a method for displaying historical audio records in a functional scenario corresponding to a schedule review is shown.

[0140] Understandably, Figure 6The process shown mainly involves the above Figure 4 The content executed in step 440 is applied in a scenario such as retrieving a schedule review of daily conversations. Figure 6 The execution subject of each step in the illustrated process may continue to be the mobile phone 100, or may be other terminal electronic devices used by the user, and no limitation is made here.

[0141] like Figure 6 As shown, the implementation process may include the following steps:

[0142] 601: The mobile phone 100 receives a search request for target content.

[0143] Exemplarily, the mobile phone 100 runs the time capsule application and detects that the user turns on the schedule review function, and can display the schedule review interface accordingly. The mobile phone 100 can receive a user operation of the user entering a search request on the schedule review interface. The user operation can be a user's voice input operation, an operation of entering text through an input method soft keyboard, etc., which are not limited here. Based on the historically stored audio data, the mobile phone 100 can provide the target content for retrieval, including but not limited to the content (such as text content) recorded corresponding to the audio data (i.e., voice), the recording location of the audio data, the recording time of the audio data, and the characters recorded in the audio data (including the voice sender including the user), etc.

[0144] As an example, refer to Figure 7a As shown, the user can click the "microphone" 711 on the schedule review interface 710 displayed on the mobile phone 100 and then speak a voice command to perform a voice input operation. For example, the user can ask a question "What did I do at location A?" The mobile phone 100 running the time capsule application can continue to execute the following steps 602 to 605 to complete the user's operation request to retrieve daily conversations. Figure 7a As shown, the user can also click the input box 712 on the schedule review interface 710 to make the mobile phone 100 display the input method soft keyboard, and then the user can input the target content to be retrieved through the input method soft keyboard. This is not limited here.

[0145] 602: The mobile phone 100 extracts search keywords corresponding to the target content to be searched.

[0146] For example, the mobile phone 100 can extract search keywords for the search request for the target content input by the user. For example, for the user's voice instruction "What did I do at location A?", the keywords "I", "location A" and "what did I do" can be extracted. Among them, "I" can be the user himself, for example, referring to the above Figure 2eThe example of "person A" or "person B". The above-extracted search keywords corresponding to the target content to be searched can be matched with the classification labels marked on the corresponding audio data when the corresponding audio data is stored.

[0147] 603: The mobile phone 100 matches the corresponding audio data based on the search keyword.

[0148] Exemplarily, the mobile phone 100 can further search for audio data with corresponding classification tags in the stored audio data based on the search keywords extracted in the above step 602. For the above example, for example, the mobile phone 100 matches one or more audio data marked with "person A" and the audio data recording location marked as "location A" based on the search keywords "I", "location A" and "what to do". Then, the mobile phone 100 can continue to perform the following step 604 based on the matched audio data to extract text content, etc.

[0149] It can be understood that the time capsule application or service used to implement the method for displaying historical audio records provided by the present application needs to process a large amount of voice information, that is, audio data. Therefore, the process of retrieving the target content from a large amount of recorded audio data needs to be fast and efficient in order to provide users with a good functional experience. Among them, as mentioned above, during the collection and storage process of audio data, the recording time, the location based on GPS positioning during recording (i.e., location information), etc. can be stored in the same memory space or associated memory space, and the mark corresponding to the above time (referred to as time mark), the label corresponding to the location (referred to as location mark), and the voiceprint mark extracted based on the audio data (corresponding to the character label), the keyword mark of the text content record (corresponding to the theme label), the content keyword mark, etc. can be added to the corresponding audio data or the path information of the audio data. In addition, when collecting audio data, the mobile phone 100 can also convert the real-time collected audio data into text content for associated storage based on the natural language recognition capability.

[0150] Based on the audio data to which corresponding tags have been added and uniformly stored, when the mobile phone 100 responds to the received search request, the strategy for extracting the search keyword for search may include, for example: splitting the search request, performing text search based on the search keyword, and performing flexible secondary search, etc. Splitting the search request means that the mobile phone 100 can split the time, place, person, and main content from the search request (such as question information) input by the user, and match them with the stored time mark, place mark, voiceprint mark, scene mark, content keyword mark, etc., and then retrieve the audio data related content with the corresponding mark.

[0151] Text search based on search keywords means: searching and matching the search keywords with the text content converted from the stored audio data. The text matching speed is fast and efficient, which is conducive to optimizing the search time. Flexible secondary search means: in some schedule review scenarios, in order to improve the search rate, the mobile phone 100 can use the audio data within a certain time period as the search object based on the preset. For example, if you ask "When was the last time I discussed Sichuan cuisine", you can set the time period for the first search to 1 month. If no search results are matched in the relevant content of the audio data within 1 month, the mobile phone 100 can perform a secondary search. Among them, the time span of the secondary search may be inconsistent with the time span involved in the search object corresponding to the first search, such as a time period of 2 months or other spans.

[0152] In other schedule review scenarios, the search keywords and search results used in the first search process can also be saved as search records. In the next search based on the same search keywords (i.e., the second search), the corresponding search records can be directly called to read the search results. Among them, the two search processes based on the same search keywords can occur within a reasonable preset time interval. After the preset time interval, the previous search record can be deleted or the time span of the audio data can be expanded when performing the second search. There is no restriction here.

[0153] 604: The mobile phone 100 obtains the converted text content corresponding to the matched audio data, the extracted voiceprint features, etc., and extracts the content matching the search keyword.

[0154] Exemplarily, the mobile phone 100 can further obtain the text content corresponding to the audio data corresponding to the above step 603, as well as the corresponding extracted voiceprint features, environmental features, health-related audio features and other multi-dimensional feature information, and extract the text content matching the search keyword and the corresponding extracted content keywords, etc. The content keywords can be, for example, a summary of the text content corresponding to the audio data or a word with a high frequency of occurrence, etc., which is not limited here.

[0155] 605: The mobile phone 100 plays or displays text or other content matching the search keyword.

[0156] Exemplarily, the mobile phone 100 can display the text and other contents matched in the above step 604 through the corresponding interface. In some embodiments, for example, the user sets the need to play the schedule review search results on the relevant interface of the time capsule application displayed on the mobile phone 100, and the mobile phone 100 can also directly play the above matched text and other contents.

[0157] In some other embodiments, if there is only one piece of audio data that matches the above step 603 executed by the mobile phone 100, the audio data that matches the above step 603 may also be played directly. No limitation is made here.

[0158] As an example, for the above example, refer to Figure 7b As shown, the mobile phone 100 can display text content 721 related to search keywords such as "me", "place A" and "what to do" on the schedule review interface 720, including "discuss project plan 1 on December 22, 2022", "discuss project plan 2 on December 23, 2022" and "have dinner with person B in a restaurant on December 24, 2022". In the embodiment of the present application, the text content 721 can be presented as Figure 7b The fishbone diagram is shown as an example. In other embodiments, the matched content displayed by the mobile phone 100 may also be presented in other forms, which are not limited here.

[0159] Furthermore, the user Figure 7b In the schedule review interface 720 shown, the text content 721 can also be clicked, including clicking "Discuss project plan 1 on December 22, 2022", "Discuss project plan 2 on December 23, 2022", and "Dining with person B in a restaurant on December 24, 2022". The mobile phone 100 can display the corresponding audio details interface, refer to Figure 7c As shown, the audio details interface 240 may, for example, display the audio data record "20231222_000000" recorded on "December 22, 2022", or display the audio data record "20231223_000000" recorded on "December 23, 2022" ( Figure 7c (not shown) or the audio data record "20231224_000000" recorded on "December 24, 2022" ( Figure 7c not shown) etc.

[0160] It is understandable that based on the above Figure 6 Combined with the above Figure 4 The implementation process of the process shown can provide users with the function of tracing past schedules and helping to quickly review historical records in the retrospective scenario of schedule review or retrieval of daily conversations based on the time capsule characteristics of audio data, which is conducive to improving the user experience of the terminal and the time capsule application.

[0161] Example 2

[0162] The embodiment of the present application provides a specific implementation process of the method for displaying historical audio records provided by the present application in a scenario where a to-do reminder needs to be set for a user.

[0163] Figure 8 According to the embodiment of the present application, a schematic diagram of the implementation process of a method for displaying historical audio records in the functional application scenario corresponding to the setting of to-do items is shown. In other embodiments, in some scenarios of voice or text and other entrustment information set to be displayed at a future time, the implementation process of setting it at a user-specified time and displaying it to a specific person (refer to the "My Entrustment" function) can also refer to the specific implementation process provided in the embodiment of the present application, which will not be repeated here.

[0164] Understandably, Figure 8 The process shown mainly involves the above Figure 4 The content executed in step 440 is applied in a scenario similar to setting to-do items for a personal secretary. Figure 8 The execution subject of each step in the illustrated process may continue to be the mobile phone 100, or may be other terminal electronic devices used by the user, and no limitation is made here.

[0165] It is understood that in some embodiments, the time capsule application can call other applications with a to-do setting function installed on the mobile phone 100, such as a calendar application, to implement the to-do setting function provided in the embodiment of the present application. In other embodiments, the to-do setting function can also be integrated into the time capsule application, which is not limited here.

[0166] This embodiment of the application takes the time capsule application calling the calendar application's set to-do function as an example to introduce in detail the specific implementation process of the historical audio recording display method provided by this application in the set to-do scenario. Figure 8 The illustrated implementation process may include the interaction between the time capsule application and the calendar application installed in the mobile phone 100. For details, please refer to the detailed description in the relevant steps, which will not be repeated here.

[0167] like Figure 8 As shown, the implementation process may include the following steps:

[0168] 801: The mobile phone 100 detects that the collected audio data involves planning or arrangement for future time.

[0169] Exemplarily, when the mobile phone 100 stores the collected audio data, the corresponding converted text content may contain future behaviors described by the user, that is, plans or arrangements for future time. For example, the user may express in some audio data "We are going to have a dinner party tomorrow", "We plan to take a short trip to location A next weekend", and "Next Monday is someone's birthday, and we need to prepare a birthday gift to wish her a happy birthday". Correspondingly, some keywords related to future behaviors can be pre-set on the mobile phone 100, such as "tomorrow", "next week", etc. in the above expressions as the basis for matching. When the mobile phone 100 detects that these keywords exist in the corresponding converted text content of the audio data, it can detect whether the corresponding audio data involves plans or arrangements for future time based on the matching results.

[0170] 802: The mobile phone 100 extracts to-do keywords related to planning or arranging for future time.

[0171] For example, the mobile phone 100 can intercept the context associated with the text content related to the audio data matching the preset keyword based on the above detection results, and extract keywords from the intercepted context content as to-do keywords. For example, for the aforementioned example "Next Monday is someone's birthday, and I need to prepare a birthday gift to wish her a happy birthday", the mobile phone 100 can extract "next Monday", "someone", "birthday", etc. as to-do keywords.

[0172] 803: The mobile phone 100 sets a to-do reminder based on the to-do keyword.

[0173] Exemplarily, the mobile phone 100 may run an application with a to-do setting function, such as a calendar application or a time capsule application integrated with this function, based on the extracted to-do keywords, and set to-do reminder items for the reminder day indicated by the to-do keywords.

[0174] It can be understood that the above-mentioned execution process of extracting to-do keywords can be completed by the time capsule application running on the mobile phone 100. The execution process of setting to-do reminders can be completed by the calendar application running on the mobile phone 100. Specifically, the time capsule application can send the extracted to-do keywords to the calendar application as input content for setting to-do reminders. Based on the received to-do keywords, the calendar application can set a to-do reminder on the corresponding reminder day using a default to-do template.

[0175] 804: The mobile phone 100 prompts the user to confirm the newly set to-do reminder related information.

[0176] Exemplarily, after completing the setting of the to-do reminder, the mobile phone 100 may display a prompt notification, prompting the user to confirm whether the reminder time, event, associated person and other related information of the set to-do reminder are accurate, whether they need to be modified, etc.

[0177] In some embodiments, the mobile phone 100 may display a prompt notification corresponding to each newly added to-do reminder. In other embodiments, the mobile phone 100 may also display a prompt notification at a preset fixed time or when the number of newly added to-do reminders meets a corresponding quantity threshold condition, to remind the user to confirm the task uniformly. This is not limited here.

[0178] It can be understood that the above-mentioned to-do reminder related information may include, but is not limited to, time dimension, location dimension, and person dimension related information. Among them, the time dimension related information may include, for example, tomorrow, X month X day, Monday / Tuesday / Wednesday / Thursday / Friday / Saturday / Sunday, etc. The location dimension may include, for example, restaurants, museums, tourist attractions, train stations, airports, etc. The person dimension may include, for example, the above Figure 2e The examples of character A, character B, etc. are not enumerated or limited here.

[0179] 805: The mobile phone 100 plays or displays the to-do items on the set to-do reminder day to remind the user.

[0180] For example, the calendar application running on the mobile phone 100 can call the information related to the upcoming reminder on the reminder day and reminder time set for each to-do item, including the relevant information of the above-mentioned time dimension, location dimension, and person dimension, and play or display it. The playback method can be, for example, a voice broadcast method, and the display method can be, for example, a pop-up window display, a status bar notification display, etc. This is not limited here.

[0181] Based on the above Figure 8 Combined with the above Figure 4 In the implementation process of the flow shown, the mobile phone 100 can provide the user with a function service of automatically setting to-do reminders based on the time capsule feature of the audio data, which is helpful for reminding the user to plan a trip, arrange a plan, etc. This is helpful for improving the user's experience of the terminal and the time capsule application.

[0182] Example 3

[0183] The embodiment of the present application provides a specific implementation process of the method for displaying historical audio records provided by the present application in a scenario where the language habits of a user or related persons need to be analyzed, such as counting the vocabulary of a user's daily expressions over a period of time.

[0184] Figure 9aAccording to the embodiment of the present application, a schematic diagram of the implementation process of a method for displaying historical audio records in a functional scenario corresponding to a behavior portrait is shown. In other embodiments, in a scenario where specific information is displayed at a specific time in a specific scenario set by the user, the process of generating specific information such as "My Quotes" for display at a specific time can also refer to the specific implementation process provided in the embodiment of the present application, which will not be repeated here.

[0185] Understandably, Figure 9a The process shown mainly involves the above Figure 4 The content executed in step 440 is applied in a scenario such as counting the behavior portraits such as catchphrases and common words in a certain period of time. Figure 9a The execution subject of each step in the illustrated process may continue to be the mobile phone 100, or may be other terminal electronic devices used by the user, and no limitation is made here.

[0186] like Figure 9a As shown, the implementation process may include the following steps:

[0187] 901: The mobile phone 100 performs word frequency analysis on the text content converted from the collected audio data.

[0188] Exemplarily, the mobile phone 100 may preset keywords for frequency analysis. The keyword vocabulary may also include vocabulary of multiple dimensions, such as "Happy New Year", "New Year's Eve", "Party", "Happy Mid-Autumn Festival", "Dragon Boat Festival", "Ankang", etc. related to festivals, "Design", "Room Type", "Plate", "Furniture", etc. related to decoration, "Forum", "College", etc. related to academic or technical, singers and song titles related to music, and TV series titles and lines, etc. In some embodiments, the above keyword vocabulary may also include some Internet hot words, catchphrases, etc., which are not limited here. It can be understood that the above word frequency analysis process can be analyzed according to different time span ranges, for example, according to time spans such as "year", "month", and "day", and the number of times the keyword vocabulary appears in the corresponding time period is analyzed. In this way, when the user views the high-frequency vocabulary in a certain time period, he can select any time span and any length of time. It can be understood that the above preset keyword vocabulary can be regularly configured and updated on the terminal such as the mobile phone 100.

[0189] Furthermore, the mobile phone 100 can match each preset keyword word from the text content converted from the audio data, and count the number of times the corresponding word appears in the text content converted from one or more audio data. The more times, the higher the word frequency corresponding to the word.

[0190] It is understood that the trigger condition for the mobile phone 100 to perform the word frequency analysis in step 901 may be based on the "behavior profile" function integrated in the time capsule application to automatically trigger the analysis of audio data within a preset time period, or may be based on an operation request input by the user. Figure 2a As shown, the user can enter the "behavior portrait" function through the scene interface 210, and input an operation request such as "what did I mainly say today" through voice, and correspondingly, the mobile phone 100 can trigger the execution of the process from step 901 to step 904 below, perform word frequency analysis and display the analysis results or recommended content related to the analysis results, etc. No limitation is made here.

[0191] 902: The mobile phone 100 determines high-frequency words in a certain period of time based on the word frequency analysis result.

[0192] Exemplarily, the mobile phone 100 analyzes that the user has said a certain word the most times in a certain period of time, or the number of times a certain word is said is ranked higher, and the corresponding words can be output as high-frequency words. For example, the user may say the word "Happy New Year" more times on January 1, and the mobile phone 100 can determine that the high-frequency word output by the user on this day is "Happy New Year". For another example, the user said "College Forum" many times during a forum on January 12, 2022, for example, up to 12 times, and the mobile phone 100 can determine that the high-frequency word output by the user on this day is "College Forum". For another example, the mobile phone 100 may analyze that the word mentioned most times by the user around January 30, 2023 is the title of a TV series, such as the County Party Committee Compound, Kuangbiao, etc. Correspondingly, it can be determined that the high-frequency words mentioned by the user during this period are the corresponding drama titles.

[0193] 903: The mobile phone 100 analyzes the user's preferences and the user's concerns within a certain period of time based on the association and combination results of the high-frequency words.

[0194] For example, in some analysis scenarios, for example, the mobile phone 100 analyzes that the user mentioned the title of a TV series and some lines in the TV series many times in December 2022, or it may just mention some lines many times. When executing steps 901 to 902 above, the mobile phone 100 can mark the corresponding lines and the title of the TV series together on the corresponding audio data, or mark the association between the two. At this time, the mobile phone 100 can combine the high-frequency lines analyzed above, and then determine that the user is more interested in a TV series during this period, or that the user is more concerned about the TV series during this period.

[0195] 904: The mobile phone 100 displays the analyzed high-frequency words and recommended contents related to the user's preferences or matters of concern to the user.

[0196] For example, in some embodiments, the mobile phone 100 may directly perform this step to display the analyzed high-frequency words after performing the above step 902 to analyze the high-frequency words. Figure 9b As shown, the mobile phone 100 can display the high-frequency words spoken by the user in the corresponding time period of the audio data recording on the mantra interface 910, for example Figure 9b As shown, "Happy New Year" corresponds to "January 1st", "College Forum" corresponds to "January 12th", and "Storm" corresponds to "January 30th".

[0197] It can be understood that if the mobile phone 100 performs word frequency analysis in step 901 above, it analyzes different time spans such as "year", "month", "day", etc., and the high-frequency vocabulary interface displayed by the mobile phone 100 can also be displayed according to the time axis of different spans. The user can click on a time span on the corresponding time axis to make the mobile phone 100 display the word analysis results within the corresponding time span. For example, if the user clicks "January" on the annual time axis, the mobile phone 100 can display Figure 9b The mantra interface 910 shown. The high-frequency words displayed on the interface 910 may be the word frequency analysis results corresponding to the time span of "January".

[0198] In some embodiments, if the mobile phone 100 obtains a large number of high-frequency words through word frequency analysis within a certain period of time, in addition to sorting and displaying these words according to their frequency of appearance within the period of time, different high-frequency words can be displayed at different times when the user views them through polling display. This is not limited here.

[0199] In other embodiments, the mobile phone 100 can obtain and display recommended content related to the user's preferences or matters of concern to the user based on the analysis results in the above step 903. For example, the mobile phone 100 can display recommended content such as episode update notifications or plot analysis short videos of a TV series based on the analysis result that the user is interested in the TV series. In other embodiments, the mobile phone 100 can also associate recommended content such as playlists and bills based on the analyzed high-frequency words, which is not limited here.

[0200] Based on the above Figure 9a Combined with the above Figure 4 In the implementation process of the flow shown, the mobile phone 100 can provide the user with a behavior profile analysis based on the time capsule features of the audio data, and can recommend content that satisfies the user's preferences or is related to the user's concerns for the user to read. This is conducive to improving the user's experience of the terminal and the time capsule application.

[0201] Example 4

[0202] The embodiment of the present application provides a specific implementation process of the method for displaying historical audio records provided by the present application in a retrospective scenario where the mood, physical and mental health status of a user or related person needs to be analyzed.

[0203] Fig.10a According to an embodiment of the present application, a schematic diagram of the implementation process of a method for displaying historical audio records in a functional scenario of emotion monitoring or health management is shown.

[0204] Understandably, Fig.10a The process shown mainly involves the above Figure 4 The application of the content executed in step 440 is shown in the scenario where emotion monitoring or health management is required. Fig.10a The execution subject of each step in the illustrated process may continue to be the mobile phone 100, or may be other terminal electronic devices used by the user, and no limitation is made here.

[0205] like Fig.10a As shown, the implementation process may include the following steps:

[0206] 1001: The mobile phone 100 extracts characteristic keywords and / or modal particles related to mood or health status based on the collected audio data.

[0207] For example, the above-mentioned characteristic keywords related to mood may be, for example, "angry", "happy", "sad", "heartbroken", "depressed", etc., and may also be keywords marked based on corresponding audio features, such as "cough", "yawn", "nasal sound when having a cold", etc., and the above-mentioned mood-related interjections may be, for example, "sigh", "haha", "hehe", etc. I will not elaborate on them here.

[0208] In some embodiments, the mobile phone 100 may also extract audio features corresponding to hoarseness in the audio data, and based on the audio features, mark the audio data with characteristic keywords related to health status, such as "cold", etc. This is not limited here.

[0209] 1002: The mobile phone 100 analyzes and determines the user's mood or health status within a corresponding time period based on characteristic keywords and / or modal particles related to the mood or health status.

[0210] For example, the mobile phone 100 can determine that the user's mood is happy in a certain period of time based on the extracted keywords such as "haha", and multiple "angry" keywords can indicate that the user's mood is average or poor, etc. The mobile phone 100 can also determine that the user is in poor health in a certain period of time based on keywords such as "cough" and "cold" marked by audio features corresponding to hoarseness.

[0211] 1003: The mobile phone 100 displays the analysis result.

[0212] For example, the mobile phone 100 can display the analysis results in text form, such as "You coughed frequently in December last year and consulted others about antipyretic drugs. You may have been infected with cold or new coronavirus". The mobile phone 100 can also display the analysis results in the form of pictures or audio and video, such as Fig.10b The mood status timeline 011 displayed on the emotion management interface 010 shown corresponds to the display format.

[0213] In some embodiments, referring to the interface of the calendar application, the mobile phone 100 can mark keywords such as "happy mood", "depressed mood" or "cold" on the corresponding audio data collection day based on the corresponding extracted audio features such as "cough" and "hoarse voice" in the audio data collected every day, and the corresponding converted text describing mood or cold-related words in the audio data, such as "I am very happy", "I am emo" and "I have a cold". In this way, when the user clicks on a certain day in history, similar to clicking on a certain date on the interface of the calendar application, he can check how the mood and physical health are on the day indicated by the date. No limitation or elaboration is made here.

[0214] In other embodiments, the mobile phone 100 may also display the associated recommended content based on the above analysis results, such as recommending some inspirational quotes to ease the user's mood, or medical advice, psychological counseling guidance, etc. This is not limited here.

[0215] Based on the above Fig.10a Combined with the above Figure 4 In the implementation process of the flow shown, the mobile phone 100 can provide users with functional services such as emotion monitoring and health management based on the time capsule features of audio data, assist users in analyzing their mood, physical and mental health status over a period of time, and can also provide users with medical advice, psychological counseling guidance, etc. This is conducive to improving the user experience of the terminal and the time capsule application.

[0216] Example 5

[0217] The embodiment of the present application provides a specific implementation process of implementing the method for displaying historical audio records provided by the present application in a retrospective scenario where meeting minutes need to be generated.

[0218] Fig.11 According to an embodiment of the present application, a schematic diagram of the implementation process of a method for displaying historical audio records in a functional scenario corresponding to meeting minutes is shown.

[0219] Understandably, Fig.11 The process shown mainly involves the above Figure 4The content executed in step 440 is shown as an application in a scenario where a meeting minutes needs to be generated for a user. Fig.11 The execution subject of each step in the illustrated process may continue to be the mobile phone 100, or may be other terminal electronic devices used by the user, and no limitation is made here.

[0220] like Fig.11 As shown, the implementation process may include the following steps:

[0221] 1101: The mobile phone 100 detects an operation request for generating meeting minutes.

[0222] 1102: The mobile phone 100 obtains audio data recording relevant content of the meeting.

[0223] 1103: The mobile phone 100 generates the meeting record or meeting minutes required by the user based on the acquired audio data.

[0224] Exemplarily, the mobile phone 100 can automatically identify the audio data including the meeting-related content based on the text content converted from the audio data, or can obtain the audio data selected by the user according to the result of the user selecting the audio data when generating the meeting record.

[0225] It is understood that the audio data can be triggered by the user opening the time capsule application on the mobile phone 100 and clicking the recording switch, or clicking the meeting recording function switch provided by the time capsule application at the beginning of the meeting to start the recording. In some embodiments, the electronic device used to record the meeting process is not limited to the above-mentioned mobile phone 100, but can also be a recording pen, a speaker, or a watch, glasses, etc. worn by the user. Among them, these electronic devices used for recording can establish a connection with the mobile phone 100 through Bluetooth or other means.

[0226] Correspondingly, during the meeting, the recording device can continuously record the voices of the user and the speakers participating in the meeting to form audio data. Furthermore, when the user instructs to generate a meeting record, the mobile phone 100 can convert the recorded audio data into a text record, and can extract the relevant audio content of the user according to the user's voiceprint features to generate the user's speech record, etc. In other embodiments, the mobile phone 100 can also generate speech records of other speakers with voiceprints and identity information recorded, which is not limited here.

[0227] Furthermore, the mobile phone can also mark the converted text content with a timestamp, and the timestamp can be associated with the corresponding playback position in the corresponding audio data for the user to listen back and trace back.

[0228] It is understandable that since some meetings may last for a long time, the corresponding recorded audio data may occupy a large storage space, so the audio data related to the meeting and the corresponding converted text records can be stored on an electronic device with a large memory space, such as a mobile phone, a personal computer (PC) or a cloud server. The voice transcription can be run on the personal recording device or on other devices selected by the user, such as a mobile phone, a PC or a cloud server.

[0229] In some embodiments, when the meeting is over, the corresponding conference management system or platform can invite each participant to provide the conference audio data recorded by the recording device they use, and share the real-time meeting records. The audio data shared by each participant can be the audio data with the best voice signal quality collected by each speaker based on the sound pickup results of the recording device they use. At this time, the conference management system that implements the time capsule feature function based on the display method of historical audio records provided by this application can, after authorization by each participant who provides the audio data, synthesize a relatively complete meeting record based on the speech records of each person extracted from the corresponding audio data and the corresponding marked timestamps. The meeting record can be a complete text record, or it can be a meeting minutes generated based on the extraction of keywords from the speech record, etc. There is no restriction here.

[0230] Therefore, it can be understood that the application for generating the meeting record or the meeting minutes can be a time capsule application or an application installed on an electronic device such as the mobile phone 100 that can interact with the time capsule application. No limitation is made here.

[0231] Based on the above Fig.11 Combined with the above Figure 4 In the implementation process of the flow shown, the mobile phone 100 can provide the user with a function service of automatically generating meeting minutes based on the time capsule feature of the audio data. This function can also be realized through the interaction between the time capsule function and some software applications such as conference systems or conference management platforms. This is conducive to improving the user experience of the terminal and the time capsule application.

[0232] Fig.12 According to an embodiment of the present application, a schematic diagram of the hardware structure of a mobile phone is shown. In other embodiments, Fig.12 The structure shown may also be the hardware structure of other terminal electronic devices, which is not limited here.

[0233] like Fig.12As shown, the mobile phone 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0234] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the mobile phone 100. In other embodiments of the present application, the mobile phone 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0235] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0236] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0237] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0238] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0239] The USB interface 130 is an interface that complies with USB standard specifications, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the mobile phone 100, and can also be used to transfer data between the mobile phone 100 and peripheral devices. It can also be used to connect headphones to play audio through the headphones.

[0240] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the mobile phone 100. In other embodiments of the present application, the mobile phone 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0241] The charging management module 140 is used to receive charging input from a charger. While the charging management module 140 is charging the battery 142 , it can also power the electronic device through the power management module 141 .

[0242] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. In some embodiments, the power management module 141 and the charging management module 140 can be provided in the same device.

[0243] The wireless communication function of the mobile phone 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0244] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in mobile phone 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0245] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the mobile phone 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc.

[0246] The wireless communication module 160 can provide wireless communication solutions for application in the mobile phone 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication technology (NFC), infrared technology (IR), etc.

[0247] In some embodiments, the antenna 1 of the mobile phone 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the mobile phone 100 can communicate with the network and other devices through wireless communication technology.

[0248] The mobile phone 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0249] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some embodiments, the mobile phone 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0250] The mobile phone 100 can realize the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194 and the application processor. The ISP is used to process the data fed back by the camera 193. In some embodiments, the ISP can be set in the camera 193.

[0251] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. In some embodiments, the mobile phone 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0252] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the mobile phone 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0253] Video codecs are used to compress or decompress digital videos. Mobile phone 100 may support one or more video codecs. Thus, mobile phone 100 may play or record videos in various coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0254] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, the intelligent cognition of the mobile phone 100 can be realized, such as: image recognition, face recognition, voice recognition, text understanding, etc.

[0255] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the mobile phone 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as music, video, and the above-mentioned recorded audio are stored in the external memory card.

[0256] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the mobile phone 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the mobile phone 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor. In an embodiment of the present application, the processor 110 can execute the above-mentioned functions of schedule review, behavior profiling, generating my quotations, setting agents, setting my instructions, emotion monitoring, and generating meeting minutes, etc., which are implemented by the time capsule feature of the mobile phone 100 based on audio data by running instructions stored in the internal memory 121, and there is no limitation here.

[0257] The mobile phone 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor.

[0258] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0259] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The mobile phone 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0260] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the mobile phone 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the ear.

[0261] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The mobile phone 100 can be provided with at least one microphone 170C. In other embodiments, the mobile phone 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the mobile phone 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0262] The earphone jack 170D is used to connect a wired earphone.

[0263] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. A capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The mobile phone 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 194, the mobile phone 100 detects the intensity of the touch operation based on the pressure sensor 180A. The mobile phone 100 can also calculate the touch position based on the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions.

[0264] The gyro sensor 180B may be used to determine the motion posture of the mobile phone 100. In some embodiments, the angular velocity of the mobile phone 100 around three axes (ie, x, y, and z axes) may be determined by the gyro sensor 180B.

[0265] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the mobile phone 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

[0266] The magnetic sensor 180D includes a Hall sensor, and the mobile phone 100 can detect the opening and closing of the flip leather case by using the magnetic sensor 180D.

[0267] The acceleration sensor 180E can detect the magnitude of the acceleration of the mobile phone 100 in various directions (generally three axes). When the mobile phone 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0268] The distance sensor 180F is used to measure the distance. The mobile phone 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the mobile phone 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.

[0269] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The mobile phone 100 emits infrared light outward through the light emitting diode. The mobile phone 100 uses the photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the mobile phone 100. When insufficient reflected light is detected, the mobile phone 100 can determine that there is no object near the mobile phone 100. The mobile phone 100 can use the proximity light sensor 180G to detect that the user holds the mobile phone 100 close to the ear to talk, so as to automatically turn off the screen to save power.

[0270] The ambient light sensor 180L is used to sense the ambient light brightness. The mobile phone 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness, and can also cooperate with the proximity light sensor 180G to detect whether the mobile phone 100 is in a pocket to prevent accidental touch.

[0271] The fingerprint sensor 180H is used to collect fingerprints. The mobile phone 100 can use the collected fingerprint characteristics to achieve fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0272] The temperature sensor 180J is used to detect temperature. In some embodiments, the mobile phone 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. In other embodiments, when the temperature is lower than another threshold, the mobile phone 100 heats the battery 142 to prevent the low temperature from causing the mobile phone 100 to shut down abnormally.

[0273] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the mobile phone 100, which is different from the position of the display screen 194.

[0274] The bone conduction sensor 180M can obtain a vibration signal. In some embodiments, the bone conduction sensor 180M can obtain a vibration signal of a vibrating bone block of a human body's vocal part. In some embodiments, the bone conduction sensor 180M can also be disposed in an earphone to form a bone conduction earphone. The audio module 170 can parse a voice signal based on the vibration signal of the vibrating bone block of the vocal part obtained by the bone conduction sensor 180M to realize a voice function.

[0275] The key 190 includes a power key, a volume key, etc. The key 190 can be a mechanical key or a touch key. The mobile phone 100 can receive key input and generate key signal input related to the user settings and function control of the mobile phone 100.

[0276] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback.

[0277] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power changes, messages, missed calls, notifications, etc.

[0278] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to or disconnected from the mobile phone 100 by inserting the SIM card interface 195 or pulling the SIM card out of the SIM card interface 195.

[0279] The following is an example of the workflow of the software and hardware of the mobile phone 100, using the scenario of recording through the Time Capsule application.

[0280] When the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into raw input events (including touch coordinates, timestamp of the touch operation, and other information). The raw input events are stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer and identifies the control corresponding to the input event. For example, if the touch operation is a touch single-click operation, and the control corresponding to the single-click operation is the control of the time capsule application icon, the time capsule application calls the interface of the application framework layer to start the time capsule application, and then starts the microphone driver by calling the kernel layer, and collects audio data through the distributed microphone array or evenly distributed microphones deployed on the mobile phone 100.

[0281] The embodiments of the present application also provide a computer program product for implementing the method for displaying historical audio records provided in the above embodiments.

[0282] The various embodiments of the mechanism disclosed in the present application can be implemented in hardware, software, firmware or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program module or module code executed on a programmable system, and the programmable system includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device and at least one output device.

[0283] A computer program module or module code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0284] Module code can be implemented with high-level modular language or object-oriented programming language to communicate with the processing system. When necessary, module code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0285] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or through other computer-readable media. Therefore, a machine-readable medium may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including, but not limited to, a floppy disk, an optical disk, an optical disk, a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic card or an optical card, a flash memory, or a tangible machine-readable memory for transmitting information (e.g., a carrier wave, an infrared signal, a digital signal, etc.) using the Internet in an electrical, optical, acoustic, or other form of propagation signal. Accordingly, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a form readable by a machine (eg, a computer).

[0286] References to "one embodiment" or "an embodiment" in the specification mean that the specific features, structures, or characteristics described in conjunction with the embodiment are included in at least one exemplary implementation or technology disclosed according to the embodiment of the present application. The appearance of the phrase "in one embodiment" in various places in the specification does not necessarily all refer to the same embodiment.

[0287] The disclosure of the embodiment of the present application also relates to an operating device for executing the text. The device can be specially constructed for the required purpose or it can include a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program can be stored in a computer-readable medium, such as, but not limited to any type of disk, including a floppy disk, an optical disk, a CD-ROM, a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an EPROM, an EEPROM, a magnetic or optical card, an application-specific integrated circuit (ASIC) or any type of medium suitable for storing electronic instructions, and each can be coupled to a computer system bus. In addition, the computer mentioned in the specification may include a single processor or may be an architecture involving multiple processors for increased computing power.

[0288] In addition, the language used in this specification has been primarily selected for readability and instructional purposes and may not be selected to describe or limit the disclosed subject matter. Therefore, the present application embodiment disclosure is intended to illustrate rather than limit the scope of the concepts discussed herein.

Claims

1. A method for displaying historical audio records, applied to a first electronic device, characterized in that: include: When a first trigger condition is detected to be satisfied at a first moment, first information is displayed, wherein the first information is determined based on first audio data collected by the second electronic device through recording, and the first trigger condition is set by the user through the second electronic device before the first moment; It is detected at a second moment that a second trigger condition is met, and second information is displayed, wherein the second information is determined based on second audio data collected by the second electronic device through recording, wherein the collection time of at least part of the audio data in the first audio data is different from the collection time of at least part of the audio data in the second audio data, and the second trigger condition is set by the user through the second electronic device before the second moment.

2. The method according to claim 1, characterized in that The method further comprises: A first user operation indicating viewing or modifying the first information is detected at a third time, and the first information is displayed, wherein the third time is earlier than the first time.

3. The method according to claim 1 or 2, characterized in that: The first trigger condition and the second trigger condition are related to at least one of the user's age status and the user's operation.

4. The method according to claim 3, characterized in that The first trigger condition includes any one of the following conditions: A condition corresponding to a trigger based on a second user operation acting on the first type of control; The first timing condition for viewing the first information set by the user satisfies the corresponding triggering condition.

5. The method according to claim 4, characterized in that The first type of controls includes at least one of the following: Indicating a first control generating the first information in a target scene; Setting a second control for displaying the first trigger condition of the first information in the target scene; and A third control that allows viewing of the first information after verifying the user's identity information in the target scenario.

6. The method according to claim 1, characterized in that A fourth time when the second electronic device collects the first audio data by recording is earlier than the first time, and a fifth time when the second electronic device collects the second audio data by recording is earlier than the second time.

7. The method according to claim 6, characterized in that The method further comprises: At the fourth moment, it is detected that a third trigger condition is satisfied, instructing the second electronic device to collect the first audio data by recording, wherein the third trigger condition is set to trigger the second electronic device to start a recording function; and At the fifth moment, it is detected that the third trigger condition is met, and the second electronic device is instructed to collect the second audio data by recording.

8. The method according to claim 7, characterized in that The third trigger condition includes any one of the following conditions: A condition corresponding to a trigger based on a third user operation acting on the second type of control to indicate starting the recording function; The second timing condition for starting the recording function set by the user satisfies the corresponding triggering condition; The location condition for enabling the recording function based on the user setting satisfies the corresponding triggering condition; The wearing conditions for turning on the recording function based on the user's settings meet the corresponding trigger conditions.

9. The method according to any one of claims 1 to 8, characterized in that The first information and the second information include any one of the following: Information of commemorative significance to the user; Information that serves as a reminder to the user; Information that has traceable effect on the user; Information that serves as a record for the user.

10. The method according to any one of claims 1 to 9, characterized in that The first electronic device and the second electronic device include any one of the following devices: Mobile phones, tablets, desktop computers, laptops, handheld computers, smart watches, smart bracelets, smart headphones, smart TVs, smart speakers.

11. An electronic device, characterized in that: include: one or more processors; One or more memories; the one or more memories store one or more programs, and when the one or more programs are executed by the one or more processors, the electronic device executes the method for displaying historical audio records according to any one of claims 1 to 10.

12. A computer-readable medium, characterized in that The readable medium stores instructions, which, when executed on a computer, cause the computer to execute the method for presenting historical audio records according to any one of claims 1 to 10.

Images