Asymmetric communication system, information processing device, information processing method, program, and terminal

Abstract

This solution addresses the asymmetrical communication barrier between users with different information processing capabilities and interfaces, particularly between children and parents, where children have difficulty with continuous speech, and parents have difficulty understanding context through intermittent messages. [Solution] The information processing device 100 receives multiple voice message data transmitted from the second terminal 300. The conversion unit 130 of the information processing device 100 concatenates the received multiple voice message data to generate a single text data. The transmission unit 140 transmits the generated single text data to the first terminal 200. The second terminal 300 allows the user to record a series of voice messages.

Description

Technical Field

[0001] The present disclosure relates to an asymmetric communication system, an information processing apparatus, an information processing method, a program, and a terminal device.

Background Art

[0002] In recent years, communication terminals for watching over family members such as children and the elderly have been becoming popular. Some of such terminals are equipped with a function that allows a user to record a voice message with a simple operation and transmit it to a smartphone of a guardian or the like. For example, Patent Document 1 discloses a system in which a user records voice with a button operation and a voice message transmitted to a remote other person is represented as text.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present disclosure has been made in view of such a situation, and an object thereof is to provide a new asymmetric communication system, an information processing apparatus, an information processing method, a program, and a terminal that alleviate the asymmetry of communication between users having different information processing capabilities and interfaces to be used and enable smoother and higher-quality information transmission.

Means for Solving the Problems

[0005] To solve the above problems, an information processing device according to one aspect of the present disclosure is an information processing device that can communicate with a first terminal and a second terminal, comprising: a receiving unit that receives a plurality of voice message data transmitted from the second terminal; a conversion unit that generates a single text data based on the plurality of voice message data received by the receiving unit; and a transmitting unit that transmits the single text data generated by the conversion unit to the first terminal. [Effects of the Invention]

[0006] According to one aspect of this disclosure, multiple intermittent voice messages can be converted into a single, context-aware text message. This reduces the burden on the recipient to understand the information and mitigates the communication asymmetry between the sender and receiver, resulting in smooth and high-quality information transmission. [Brief explanation of the drawing]

[0007] [Figure 1] This figure shows the overall configuration of an asymmetric communication system according to one embodiment of the present disclosure. [Figure 2] This is a block diagram showing an example of the hardware configuration of the information processing device according to this embodiment. [Figure 3] A block diagram showing an example of the hardware configuration of the first terminal according to this embodiment. [Figure 4] A block diagram showing an example of the hardware configuration of the second terminal according to this embodiment. [Figure 5] This block diagram shows an example of the functional configuration of the first terminal according to this embodiment. [Figure 6] This block diagram shows an example of the functional configuration of the information processing apparatus according to this embodiment. [Figure 7] A block diagram showing an example of the functional configuration of the second terminal according to this embodiment. [Figure 8] This flowchart shows an example of the contextualization process for multiple voice messages performed by the information processing device according to this embodiment. [Figure 9]This figure shows an example of UI screen transitions for the second terminal according to this embodiment. [Figure 10] This figure shows an example of the data structure of the message management table used in this embodiment. [Figure 11] This figure shows an example of the display screen of the first terminal according to this embodiment. [Figure 12] This flowchart shows an example of the flow of voice message recording and transmission processing performed by the second terminal according to this embodiment. [Figure 13] This flowchart shows an example of the flow of receiving and displaying text data, etc., performed by the first terminal according to this embodiment. [Modes for carrying out the invention]

[0008] The embodiments of this disclosure will be described in detail below with reference to the drawings. In each drawing, the same components are denoted by the same reference numerals, and redundant explanations are omitted.

[0009] Figure 1 shows the overall configuration of an asymmetric communication system 1 according to one embodiment of the present disclosure. The concept of this system 1 is to convert and translate intermittent voice messages from a user P2 (hereinafter referred to as the second user), such as a child, who can only provide simple input, into a single context that is easy for a user P1 (hereinafter referred to as the first user), such as a parent, to understand, using the intelligence of the information processing device 100, and deliver it to the user P1 (hereinafter referred to as the first user).

[0010] As shown in Figure 1, the asymmetric communication system 1 according to this embodiment is configured as a monitoring system that monitors the location information of users. System 1 mainly comprises a first terminal 200 (a terminal for the caregiver) used by a first user P1 (for example, a caregiver), a second terminal 300 (a terminal for the person being cared for) used by a second user P2 (for example, a child or other person being cared for), and an information processing device 100 (a server) that is connected to these terminals in a way that allows communication. The first terminal 200, the second terminal 300, and the information processing device 100 send and receive data to and from each other via a network NW such as the Internet. The information processing device 100 may be implemented as a standalone server device, but it can also be distributed and implemented on a cloud infrastructure that can scale according to the number of users and load conditions. In addition to communication between the first terminal 200 and the second terminal 300, it is also possible to connect to an administrator interface such as a Web UI or dedicated app for settings and notification confirmation by parents.

[0011] The first terminal 200 is a relatively high-function information processing terminal, such as a smartphone, tablet, personal computer (PC), or smartwatch. The second terminal 300 is a small, dedicated device with intentionally limited functionality, carried by children or the elderly. The information processing device 100 may be a single server computer or may consist of multiple computers built in a cloud computing environment.

[0012] Figures 2 to 4 are block diagrams showing examples of the hardware configuration of each device according to this embodiment.

[0013] Figure 2 shows an example of the hardware configuration of the information processing device 100 according to this embodiment. The information processing device 100 includes a processor 101, a main memory 102, an auxiliary memory 103, and a communication interface 104, etc.

[0014] FIG. 3 shows an example of the hardware configuration of the first terminal 200 according to the present embodiment. The first terminal 200 includes a processor 201, a main storage device 202, an auxiliary storage device 203, a communication interface 204, a display 205, an input device 206, and the like.

[0015] FIG. 4 shows an example of the hardware configuration of the second terminal 300 according to the present embodiment. The second terminal 300 includes a processor 301, a main storage device 302, an auxiliary storage device 303, a communication interface 304, a display 305, an input device 306, a microphone 307, a speaker 308, a GPS receiver 309, an acceleration sensor 311, and the like.

[0016] FIG. 5 is a block diagram showing an example of the functional configuration of the first terminal 200 according to the present embodiment. The control unit 210 of the first terminal 200 is realized by the processor 201 executing a predetermined application program, and functions as a receiving unit 220, a display control unit 230, and a geofence setting unit 240.

[0017] The receiving unit 220 receives single text data, position information, various notifications described later, and the like from the information processing device 100.

[0018] The display control unit 230 displays the single text data and position information received by the receiving unit 220 on the display 205 in an associated manner. This display control has the advantage that the first user can intuitively grasp the message content and the situation of the transmission location on one screen.

[0019] Specifically, a display mode such as displaying a text message like an information card and displaying a marker indicating a map and a position within the card can be considered.

[0020] Further, when the received message is generated by integrating a plurality of voice messages, the display control unit 230 may display an integration mark indicating that on the display 205.

[0021] Furthermore, if the confidence level of speech recognition is low and multiple candidate text data are sent from the server, these candidates will be displayed in a list near the main text message. This feature allows parents to supplement their judgment by using their own knowledge to determine the most likely word.

[0022] The geofence setting unit 240 accepts input from the first user (parent) and has the function of setting a specific geographic area on the map as a "geofence." For example, it sets a "living area" including schools, cram schools, parks, etc. as a geofence and transmits that information to the information processing device 100.

[0023] Figure 13 is a flowchart showing an example of the flow of receiving and displaying text data and the like, performed by the first terminal 200 according to this embodiment.

[0024] First, the first terminal 200 waits to receive data from the information processing device 100 (step S300). When data is received (Yes in S300), the type of received data is determined (step S302).

[0025] If the received data is a regular message containing a single text data (regular message in S302), the display control unit 230 extracts the text data and associated location information and displays them on the display 205 (step S304). At this time, if there is identification information indicating that it is an integrated message, the integrated mark is also displayed.

[0026] If the received data contains multiple candidate text data (a message with candidates in S302), the display control unit 230 displays the primary candidate text data as usual, and also displays the other candidate texts in a list in the nearby candidate display area (step S306).

[0027] If the received data is a special notification such as an emergency notification (SOS) (special notification in S302), the display control unit 230 will display a predetermined warning or play an alert sound that is different from the standard message display (step S308).

[0028] Once the display process is complete, the system returns to the data reception waiting state (step S300).

[0029] Figure 7 is a block diagram showing an example of the functional configuration of the second terminal 300 according to this embodiment. The second terminal 300 includes a control unit 310 that controls its overall operation. This control unit 310 operates as multiple functional units in cooperation with various hardware components when the processor 301 executes a predetermined program. Specifically, the control unit 310 includes the functions of an input receiving unit that accepts user operations from an input device 306, a transmitting unit and a receiving unit that exchange data with the information processing device 100 via a communication interface 304, a storage device control unit that manages the writing and reading of data to and from the transmission queue 331 and the present buffer 332 provided in the auxiliary storage device 303, and a notification unit that provides visual and auditory feedback to the user using a display 305 and a speaker 308. The second terminal 300 may also be configured to be equipped with a dedicated SIM card corresponding to the communication interface 304 and to communicate directly with the information processing device 100 using a cellular network such as LTE or LTE-M. This configuration allows the second terminal 300 to connect to the internet independently, without relying on the guardian's smartphone or other devices. This ensures a stable communication environment even while on the go or away from home, making it particularly effective in situations requiring real-time information, such as emergency notifications. Furthermore, the communication interface 304 of the second terminal 300 may be implemented using low-power communication methods such as Wi-Fi, Bluetooth LE (BLE), or LoRaWAN. In particular, for terminals carried constantly by children or the elderly, it is desirable to have a configuration that switches between multiple methods depending on the communication frequency and distance.

[0030] The second terminal 300 has a recording button as part of the input device 306 for the user to record a voice message. In this embodiment, the second terminal 300 may have an upper limit (e.g., 40 seconds) on the recording time of a single voice message due to technical or operational constraints.

[0031] When there is a limit to the recording time, if what the child wants to convey exceeds the time limit, the user will inevitably have to split the message into multiple parts.

[0032] The recording button controls the start and end of audio recording by detecting a user press or long press. The specific form of the recording button can vary; for example, it may be a physical push button, or it may have a capacitive touch sensor on its surface.

[0033] Furthermore, if the display 305 is a touch panel, it is also possible to implement a button on the graphical user interface (GUI) that functions as a recording button by allocating a portion of its display area to this function.

[0034] Children and other young users sometimes have difficulty coordinating the timing of pressing the record button with the start of their speech, resulting in the beginning of their speech not being recorded. Additionally, users may add forgotten words immediately after releasing the record button, but this end of speech was not recorded on previous devices.

[0035] To address these issues, the control unit 310 of this embodiment may perform a pre-recording function and a follow-up recording function. Furthermore, if the recording is too short or silent, a failure message or an animation prompting re-recording can be displayed on the display 305, making it easier for the user to recognize the recording content. Additionally, the parent's first terminal 200 can also implement a function to display the audio waveform and playback icon of the received message, allowing for audio reconfirmation and correction requests.

[0036] In the pre-sensing recording function, if the recording button is equipped with a touch sensor, the system starts pre-recording audio to the pre-sensing buffer 332 provided in the auxiliary storage device 303 when the user touches the button (touch detection).

[0037] Subsequently, when the control unit 310 detects that the user has pressed a button (push detection), it reads audio data from the present buffer 332 starting from a predetermined time (for example, 1 to 2 seconds) prior to the push detection and adds it to the transmit queue 331. This makes it possible to reliably acquire the beginning portion of the utterance.

[0038] The follow-up recording function continues recording for a predetermined time (e.g., 1 second) even after the user has released their finger from the recording button (detection of the end of the push), as long as the touch sensor on the display 305 continues to detect contact from the user's finger. This makes it possible to more reliably capture the end of the utterance.

[0039] The control unit 310 temporarily stores the recorded audio data in a transmission queue 331 provided in the auxiliary storage device 303 to allow the user to instruct the recording of continuous voice messages. This continuous recording and transmission function is particularly effective in reducing stress and not interrupting the user's thought process, especially in situations where there is a limit on the recording time as described above.

[0040] Furthermore, the control unit 310 monitors the status of the transmission queue 331 and the transmission status of the communication interface 304, and displays one of several predetermined different display modes on the display 305 according to their status.

[0041] For example, a first display mode indicating a waiting state for transmission (e.g., a bouncing animation) and a second display mode indicating the start of transmission (e.g., a flying animation) are used to differentiate between the two. This visual feedback allows users to intuitively understand that their actions have been accepted by the system and are being processed appropriately.

[0042] In particular, using positive language to indicate the "waiting to send" status reduces the anxiety and stress users feel about waiting times, and helps maintain their motivation to communicate. This contributes not only to the technical benefit of improved visibility, but also to an improved user experience and, ultimately, to improved accessibility.

[0043] Figure 9 shows an example of UI screen transitions of the second terminal 300 according to this embodiment, which realizes such visual feedback. As shown in Figure 9, the second terminal 300 has a rounded casing that is easy for children to hold, and an input device 306 (record button) for performing the main operation is located in the center, with a display 305 for displaying the status located above it. The UI screen transitions follow different paths depending on the message recording method. First, when the user records a single message, the screen transitions from the standby state (A) to the recording state (B). If communication can be started immediately after the recording is finished, the screen does not go through the "waiting to send" state (C) but immediately transitions to the sending state (D). In this state (D), as a second display mode, an animation is displayed, for example, in which an icon resembling a message flies into the sky. Therefore, in the case of single message recording, the screen transition mainly follows the path (A) → (B) → (D). On the other hand, if a user records multiple messages consecutively and messages accumulate in the send queue 331, the "Waiting to Send" state (C) is displayed. In this state, as the first display mode, a positive animation is displayed, such as a bouncing message icon, allowing the user to understand that sending is pending while reducing the stress of waiting. After sending begins, the system transitions to state (D). Therefore, if continuous recording occurs, the screen transitions follow the path (A)→(B)→(C)→(D). By visually distinguishing these two transition paths, users, especially children, can intuitively and enjoyably understand the communication status of the device.

[0044] Figure 12 is a flowchart showing an example of the flow of voice message recording and transmission processing performed by the second terminal 300 according to this embodiment.

[0045] The second terminal 300 first waits for user input to the recording button (step S200). When contact with the recording button is detected (Yes in S202), the control unit 310 starts pre-sense recording and temporarily stores the audio in the pre-sense buffer (step S204).

[0046] Next, the device waits for a button press (step S206), and when a press is detected, it starts recording (step S208). When the user releases their finger from the button, the end of the press is detected (Yes in step S210), and the device stops recording.

[0047] Next, the control unit 310 reads audio data for a predetermined time period prior to the start of main unit recording from the presence buffer and combines it with the main unit recording data (step S212). Furthermore, it determines whether the conditions for follow recording (for example, contact continues after the button is pressed) are met (step S214), and if they are met, it performs follow recording for a predetermined time and combines it (step S216).

[0048] The single voice message data thus generated is added to the transmission queue (step S218). The transmission queue is sent asynchronously to the information processing device 100 according to the status of the communication unit. Once processing is complete, the terminal returns to the operation waiting state (step S200).

[0049] Figure 6 is a block diagram showing an example of the functional configuration of the information processing device 100 according to this embodiment. The control unit 110 of the information processing device 100 functions as a receiving unit 120, a conversion unit 130, a transmitting unit 140, a location information management unit 150, a geofence determination unit 160, and a notification control unit 170.

[0050] The conversion unit 130 is one of the core functions of the present invention and has the function of generating a single text data based on multiple received voice message data. To achieve this, the conversion unit 130 performs voice message integration processing, followed by voice recognition processing and value-added processing.

[0051] The voice message integration process is mainly performed by the message group identification unit 131 and the integration processing unit 132. The message group identification unit 131 identifies multiple voice message data received consecutively from the same second terminal 300 within a predetermined time as a message group consisting of a "series of utterances".

[0052] The integrated processing unit 132 concatenates the identified message group while maintaining the reception order, and generates a single concatenated audio data.

[0053] This voice message integration process is particularly useful for users whose speech is unclear or who can only pronounce words syllable by syllable. For example, if a user wants to say "car," it might be sent as three short voice messages: "ku," "ru," and "ma."

[0054] In such cases, it is extremely difficult for the speech recognition engine to identify a word based on the individual audio data "ku" alone. However, in this disclosure, these three audio data are concatenated into a single audio piece called "kuruma" before undergoing speech recognition processing.

[0055] This has the remarkable effect of supplementing the context of the utterance, dramatically improving the likelihood of it being correctly recognized as a word.

[0056] Furthermore, when the integrated processing unit 132 concatenates multiple voice messages, it can estimate the contextual connection between messages with greater accuracy by analyzing the follow-up recording portion of a given message and the pre-recording portion of the next received message.

[0057] For example, if the audio at the end of the follow-up recording and the audio at the beginning of the present recording are acoustically continuous, these two messages can be considered strongly linked semantically. Based on this judgment, it becomes possible to connect them seamlessly during text conversion or to insert more natural commas. This improves contextual understanding in audio message integration processing.

[0058] The value-added processing unit 134 performs processing to add value to the generated text data. Specifically, it performs processing to interpret silent intervals between messages and automatically insert commas and other punctuation marks into the text, and processing to analyze emotions from the acoustic characteristics of audio data and automatically add corresponding emojis to the text. Furthermore, if the text data generated by the integration exceeds a predetermined number of characters, the value-added processing unit 134 may perform processing to summarize its content. This makes it possible for the first user P1 to quickly grasp the main points even from lengthy reports.

[0059] The speech recognition unit 133 generates text data based on integrated or value-added processed speech data.

[0060] The speech recognition model used by the speech recognition unit 133 may include not only a general-purpose model but also a personalized language model learned from the communication history of a specific user group. Furthermore, the user may be able to select from multiple speech recognition models, such as "standard," "children's," and "specific theme," on the settings screen of the first terminal 200. Additionally, if the utterance from the second terminal 300 contains new vocabulary or context, the information processing device 100 can trigger automatic retraining of the personalized model under certain conditions.

[0061] This personalized language model addresses the problem that general-purpose models may misrecognize slang that is temporarily popular among children or nicknames used only within families. By using a model optimized for an individual, recognition accuracy is improved, leading to smoother communication.

[0062] Figure 8 is a flowchart showing an example of the flow of contextualization processing of multiple voice messages performed by the information processing device 100 according to this embodiment.

[0063] First, the information processing device 100 receives voice message data from the second terminal 300 (step S100). Next, it determines whether new message data has been received from the same terminal ID within a predetermined time (step S102).

[0064] Here, the message group identification unit 131 may determine whether or not to perform integrated processing by taking into account not only temporal conditions but also other information. For example, it is conceivable that the unit may determine that the user is moving based on changes in the location information of the second terminal 300 managed by the location information management unit 150.

[0065] Furthermore, if integration processing is not performed (for example, if the relevance between voice messages is determined to be below a threshold), the received voice message data may be processed as separate text data and sent sequentially to one terminal. This improves processing flexibility and prevents misunderstandings caused by unnecessary integration.

[0066] If the user is moving, it may be estimated that their speech is likely to be interrupted, and the system may be controlled to perform integrated processing. Similarly, if the system determines from the data of the accelerometer 311 that the user is walking, or if the system determines from the location information that the user is in a noisy outdoor environment, integrated processing can also be triggered.

[0067] If no new message is received after a predetermined time has elapsed (No in S102), processing of the accumulated message group begins. The information processing device 100 concatenates multiple voice message data from the same group in the order they were received (step S104).

[0068] Next, value-added processing is performed on the concatenated audio data (step S106). Then, speech recognition processing is performed to generate single text data (step S108).

[0069] Finally, the generated single text data and associated location information are sent to the first terminal 200 (step S110), and the process ends.

[0070] Furthermore, upon receiving the first audio data in the message group, the transmitting unit 140 may first send a preliminary text message containing only that data to the first terminal 200. Subsequently, if subsequent audio data is received within a predetermined time, the transmitting unit 140 will send a final text message integrating all of that data. This configuration allows the recipient (first user P1) to immediately know that a message has been received and to ultimately understand the content in context, thus achieving both real-time functionality and accuracy.

[0071] (An example of application for the second terminal) As an example of the application of this disclosure, we will describe in detail the characteristics of a user P2 of the second terminal 300, for example, a child in the lower grades of elementary school, and the desirable physical and functional features and limitations of the second terminal 300 that result from this.

[0072] In this application example, it is desirable that the second terminal 300 has a physical configuration that takes into account the physical and cognitive characteristics of children.

[0073] Firstly, the shape and material of the casing are important. It is preferable that the entire device be small and lightweight so that even small hands can hold it securely. Furthermore, rounded corners enhance safety in the event of an accidental collision.

[0074] For the casing material, for example, non-toxic ABS resin that is safe even if ingested, or silicone material that absorbs shock and provides a soft, non-slip feel may be used.

[0075] Secondly, durability is important. Children are likely to drop the device or get it wet. Therefore, it is desirable that the second device 300 has high impact resistance.

[0076] Furthermore, its IPX7 waterproof rating and IP6X dustproof rating reduce the risk of malfunction when used outdoors in rainy weather or in places like sandboxes in parks.

[0077] Thirdly, there is the operating interface. Because children's ability to control fine finger movements is still developing, they may have difficulty accurately pressing small buttons. Therefore, it is preferable that the physical operating buttons provided as input device 306 are intentionally designed to be larger than those on typical smartphones used by adults.

[0078] Furthermore, to avoid confusion during operation, it is desirable that the total number of buttons be limited to three or fewer, including the voice recording button and the SOS button described later.

[0079] Next, we will describe the desirable functional features and limitations of the second terminal 300 in this application. Some of these features are not directly related to the core voice message processing of this disclosure, but are important in defining the specific technical context to which this disclosure applies.

[0080] Firstly, there is an emergency notification function (SOS function). The second terminal 300 may be equipped with a dedicated SOS button that is distinguished from other buttons in shape, color, and placement. If this SOS button is pressed and held for a predetermined time (for example, 3 seconds), the second terminal 300 forcibly sends a notification that an emergency has occurred, along with its current location information, to all pre-registered first terminals 200.

[0081] Secondly, there is the parental control function. Parents can remotely manage the operation of the second device 300 from a dedicated application on the first device 200.

[0082] For example, there are features that allow you to set a silent time when sending and receiving voice messages is not permitted, such as during school classes, and features that limit who you can communicate with to pre-approved contacts, such as family members.

[0083] Thirdly, the display interface is simplified. The display 305 of the second terminal 300 does not display complex menus or settings screens, but is specialized in displaying only the minimum necessary status icons such as battery level and signal strength, and an animation indicating the message transmission status, which is a feature of the present invention.

[0084] The physical and functional characteristics and limitations described so far are not merely additional features. These limitations are precisely what highlight the technical significance and non-obviousness of the server-side voice message integration processing, which is the core of this disclosure.

[0085] For example, due to the functional constraint of not having a text input interface, the user's means of expression is limited to voice. Furthermore, due to the limitation on the recording time per session, long conversations inevitably have to be divided and transmitted in segments.

[0086] Furthermore, due to the user's characteristic of being a child who is easily distracted, their speech tends to be inherently fragmented.

[0087] Thus, the second terminal 300, by its design philosophy, is intentionally designed to easily generate "discontinuous and short audio data." Precisely under these unique technical conditions, the process by which the information processing device 100 collects, connects, and contextualizes these fragmented audio data, supplementing nuances such as emotion and pauses to reconstruct them as meaningful text information, goes beyond mere convenience.

[0088] This will be an essential and indispensable solution for enabling communication within this system.

[0089] Therefore, unlike general-purpose speech recognition systems, this disclosure organically combines a client-side UI with server-side information processing to solve specific challenges arising under particular constraints. This idea is not readily apparent to those skilled in the art and constitutes a strong basis for the inventive step of this disclosure.

[0090] (Example of an architectural variation) This disclosure is not limited to the embodiments and applications described above, and various modifications are possible without departing from its essence. Furthermore, elements of each embodiment may be combined as appropriate.

[0091] The arrangement of the functional blocks in this disclosure is not limited to the embodiments described above. Within a technically reasonable scope, each functional block can be arranged in various ways in each device constituting the system (information processing device 100, first terminal 200, second terminal 300).

[0092] As a variation 1, a client-centric configuration can be considered in which the first terminal 200 is responsible for the main conversion processing. Multiple voice message data transmitted from the second terminal 300 are simply relayed and transferred to the first terminal 200 without being integrated or converted by the information processing device 100.

[0093] The first terminal 200 performs a series of transformation processes based on the received multiple voice message data, including identifying the message group, concatenating them, speech recognition, and value-added processing, to generate a single text data and display it on its own display.

[0094] This configuration has the advantage of reducing the processing load on the information processing device 100 and increasing the system's scalability. Furthermore, it is advantageous from a privacy perspective because sensitive information, such as the user's voice data, can be processed within the first terminal 200 under the user's control without being transmitted to a server.

[0095] In particular, by adopting a configuration in which a personalized language model is stored within the first terminal 200 and the model is updated using voice data received within the first terminal 200, it is possible to continuously improve recognition accuracy while suppressing the amount of data transmitted and ensuring a high level of privacy.

[0096] As a second variation, a distributed processing configuration can be considered in which functions related to the conversion process are distributed among the devices. For example, the second terminal 300 performs preprocessing such as noise reduction and removal of silent sections on the audio data acquired by the microphone 307 before transmitting it to the information processing device 100.

[0097] The information processing device 100 is dedicated to performing integrated processing of voice messages and speech recognition processing that require computing resources, and transmits the text data of the recognition results to the first terminal 200.

[0098] The first terminal 200 performs value-added processing on the received text data, such as adding emojis to indicate emotions and controlling the display of multiple recognition candidates, as well as final display processing. By distributing the processing according to the characteristics of each device in this way, the responsiveness of the entire system can be improved and resources can be used efficiently.

[0099] As a third variation, a high-performance configuration of the second terminal 300 is conceivable, in which the second terminal 300 has advanced processing capabilities. The second terminal 300 concatenates multiple recorded voice message data using its own processor 301, performs voice recognition processing, and generates text data.

[0100] The generated text data is then sent to the information processing device 100. The information processing device 100 links the received text data with the separately received location information and sends it to the first terminal 200.

[0101] This configuration has the advantage of significantly reducing network bandwidth consumption because it transmits text data instead of voice data. This is particularly effective for use in locations with unstable communication environments or when it is necessary to reduce communication costs.

[0102] (Example of functional integration) Each functional unit described in this embodiment can be configured to provide further technical advantages by coordinating with other functional units and information.

[0103] When determining whether or not to integrate multiple voice messages, the message group identification unit 131 may use information from various sensors installed in the second terminal 300, in addition to temporal proximity and location information.

[0104] For example, consider a case where, based on data acquired from the accelerometer 311, it is determined that the second user P2 is in a physically active state, such as while running or playing on playground equipment. In such situations, speech tends to be inherently interrupted, and silent intervals between messages tend to be longer. The activity state of the second user is estimated from sensor output and changes in location information, and based on the estimated activity state, a decision is made as to whether to identify multiple voice messages as a "series of utterances" and perform integration processing. If integration processing is decided upon, for example, the voices "Today," and "At the park, I rode the slide!" are concatenated, and speech recognition and value-added processing are performed. Furthermore, the high tone of the last utterance suggests an emotion of "joy."

[0105] Furthermore, a configuration that suppresses integration processing is conceivable if it is determined that integration is "not necessary" based on information from an accelerometer or other sensors. For example, if audio is transmitted during a long period of static conditions rather than a physically active situation, it is highly likely that the content is unrelated, and individual processing may be more appropriate.

[0106] Therefore, when an active state is detected, the message group identification unit 131 dynamically extends the time threshold (predetermined time) for identifying the message group or relaxes the conditions for integration determination.

[0107] This results in the technical benefit of improved text accuracy, as appropriate integration processing is performed according to the user's situation, and unnatural fragmentation of context is suppressed. Similarly, if the ambient noise level of the audio collected by microphone 307 is analyzed and it is determined that the environment is noisy (e.g., outdoors, train platform), similar control may be applied.

[0108] The value-added processing unit 134 may use information about the geographical location of the second terminal 300, provided by the geofence determination unit 160, to perform processing that adds contextual value to the generated text data.

[0109] The geofence determination unit may be configured to skip value-added processing if it does not match the relevant geographical area (for example, if it is determined to be outside the geofence). Furthermore, if location information cannot be obtained, processing can be performed based on pre-configured default completion rules.

[0110] For example, suppose that the second terminal 300 is located within a geofence that has been pre-configured as "school" by the geofence setting unit 240, and the text "arrived" is generated from the voice message.

[0111] At this time, the value-added processing unit 134 interprets the context of the location information and generates text that complements the information, such as "I've arrived at school" instead of just "I've arrived," or automatically adds an emoji representing the school.

[0112] This allows the first user P1 to understand the text content more intuitively and accurately, improving the quality of communication.

[0113] Furthermore, it is possible to control the speech recognition unit 133 by triggering a combination of a specific geofence (e.g., "cram school") and a time period (e.g., 5pm to 7pm) to temporarily switch the language model it uses to a model that prioritizes the recognition of specialized terminology related to learning. This makes it possible to selectively improve speech recognition accuracy under specific circumstances.

[0114] The message group identification unit 131 may identify the message group to be integrated based on semantic relationships obtained from the content of the voice messages themselves, in addition to or independently of temporal conditions and sensor information.

[0115] In this configuration, the information processing device 100 first performs a preliminary speech recognition process on each of the consecutive voice message data received from the second terminal 300 to generate provisional text data. The accuracy of the text at this stage does not need to be perfect.

[0116] Next, the message group identification unit 131 analyzes the semantic relationships between the multiple generated provisional text data. Possible methods for analyzing relationships include determining the degree of overlap of keywords such as nouns and verbs between the text data, performing coordinating analysis to determine whether pronouns (e.g., "it," "that") refer to a specific object in the previous text data, or calculating the degree of topic agreement or semantic vector similarity using natural language processing techniques.

[0117] If the analysis determines that the semantic relevance between text data is higher than a predetermined threshold, the group of audio messages from which they originated will be considered a "series of contexts" and selected for integration processing, even if the reception interval between each audio message exceeds the normal temporal threshold. This method makes it possible to accurately capture a series of utterances that continue on a topic even if they are separated in time, and to generate more natural and contextually relevant text information.

[0118] (Other application examples) The system disclosed herein is applicable not only to communication between parents and children, but also to communication between diverse users where there are asymmetries in information processing capabilities and user interfaces.

[0119] One possible application example is supporting communication between elderly people and their families. In this case, the second user P2 is an elderly person, and the second terminal 300 may take the form of a pendant, wristwatch, or be integrated into a cane so that it is easy for the elderly person to carry with them at all times.

[0120] The control buttons should be large and easy to press, and in addition to a dedicated SOS button for calling for help in emergencies, there may also be a button for calling a specific family member.

[0121] The conversion unit 130 of the information processing device 100 performs integrated processing that takes into account the slow speaking style and the characteristics of speech that are unique to the elderly, such as long pauses in the middle of utterances. For example, it converts multiple intermittent utterances such as "I took my medicine... um... yes, I took it" into a single concise message, "I took my medicine."

[0122] Furthermore, the value-added processing unit 134 may include a health condition monitoring unit (not shown) that analyzes the acoustic characteristics of the voice data, such as tone, volume, and speaking speed, over the long term and detects differences from normal conditions.

[0123] If changes such as a lack of energy in the voice or slow responses are detected, an alert message such as "※Your voice tone seems lower than usual" is added to the text message and sent to the first terminal 200. This allows family members to detect changes in the health of elderly individuals early through daily communication.

[0124] As an example of application, field worker support in business settings is conceivable. This system can be applied to business applications that require hands-free reporting, such as construction sites, logistics warehouses, security, and delivery services. In this case, the second user P2 is a field worker, and the second terminal 300 is configured as a highly robust, dustproof, and waterproof device that can be attached to a helmet or work clothes.

[0125] The input device 306 features physical buttons that can be reliably operated even while wearing gloves, as well as a voice activation function using a specific wake word (e.g., "Start Report").

[0126] The speech recognition unit 133 of the information processing device 100 uses a language model specialized in industry-specific jargon and abbreviations. The conversion unit 130 receives multiple fragmented voice reports from workers, such as "Material A, delivery complete," "Point B, no abnormalities," and "Proceed to the next patrol," and integrates them.

[0127] Furthermore, the value-added processing unit 134 not only converts the data to text, but also generates structured text data such as "(Work Report) Time: 14:32, Location: Point B, Content: No abnormalities, Next action: Proceed to the next patrol" according to a predetermined report format.

[0128] The generated data is not only sent to the first terminal 200 (the administrator's PC or smartphone), but is also automatically linked to external business systems such as work report systems and project management tools via API (Application Programming Interface). This streamlines reporting tasks and enables real-time sharing and recording of information.

[0129] As an example of application, language learner support can be considered. This system can also be used as a tool for foreign language learners to improve their speaking ability. In this case, the second user P2 is the language learner, and the first user P1 is the teacher or the learner themselves. The second terminal 300 has a function to set the language that the learner wants to practice (e.g., English, Chinese).

[0130] When the learner intermittently utters words or short phrases they are practicing, the information processing device 100 integrates them into a series of utterances. The speech recognition unit 133 converts the learner's utterances into text and simultaneously detects pronunciation and grammatical errors.

[0131] The conversion unit 130 generates the original utterance text, the corrected sentence text, and the error notification (e.g., "The pronunciation of 'l' and 'r' is unclear") as a single text data.

[0132] Furthermore, it generates audio data of correct pronunciation by native speakers and sends it along with text data to the first terminal 200 (a smartphone app for learners or a management screen for teachers).

[0133] Learners can efficiently practice speaking by objectively reviewing how their utterances are perceived and how they should be corrected. Teachers can centrally review the practice history of multiple learners and use this information to provide individualized instruction.

[0134] (Example 1) In this embodiment, we assume a scenario in which a child (second user P2) reports what happened while playing in the park to their guardian (first user P1). The child has a second device 300, and the guardian has a first device 200 (smartphone).

[0135] The child presses the recording button on the second terminal 300, says "Today," and releases the button. The control unit 310 adds this voice data to the transmission queue 331. The communication interface 304 immediately starts transmitting, and the display 305 displays a second display mode (e.g., the movement of a letter flying away).

[0136] Without waiting for the message to be sent, the child presses the button again and says, "I went down the slide at the park!" If the first message has not yet been sent, the display 305 shows the first display mode (e.g., a letter bouncing), allowing the child to happily recognize that their message is waiting.

[0137] The information processing device 100 receives these two voice messages within a predetermined time and identifies them as a "series of utterances." It then concatenates the voices "Today," and "I went down the slide at the park!" and performs voice recognition and value-added processing. The high tone of the last utterance suggests an emotion of "joy."

[0138] As a result, a single text message and location information are displayed on the parent's first device 200. Figure 11 shows an example of the display screen 400 of the first device 200 at this time. The display screen 400 shows a chat-style UI, and a speech bubble 401 is displayed along with the child's icon.

[0139] Next to speech bubble 401, a merger mark 405 (for example, a chain icon) is displayed to indicate that this message has been merged from multiple audio sources. Inside speech bubble 401, the text message 402, "Today, I went down the slide at the park!" is displayed at the top, and below it, a thumbnail 403 of a map showing the area around the park is displayed.

[0140] (Example 2) In this embodiment, we assume a scenario where a child wants to say the name of the vehicle "Shinkansen," but their pronunciation is unclear and they send three short voice messages like "Shin, Kan, Ten."

[0141] The information processing device 100 combines these three audio data through audio message integration processing and processes them as a single audio message, "shinkanten," for recognition. The speech recognition unit 133 determines that the confidence level of the recognition result for this audio is low and, while selecting "Shinkansen" as the most likely candidate, generates multiple candidate text data such as "shinkanten?" and "kanten?".

[0142] As a result, on the display screen 400 of the parent's first device 200, the main candidate "Shinkansen" is displayed in the speech bubble 401, and other possibilities such as "Shinkanten?" and "Kanten?" are displayed in a list in the nearby candidate display area 406.

[0143] Parents know that their child has recently become fascinated with "bullet trains," so they can easily determine that the primary candidate is correct. This allows for smooth communication even with utterances that would be difficult to identify using voice recognition alone, by leveraging the parents' knowledge.

[0144] (Example 3) This embodiment assumes a scenario in which a second user P2 (child) sends messages to a first user P1 (parent) while playing on a trampoline in a park. The child intermittently sends three short voice messages while jumping: "Right now," "I'm jumping," and "It's fun!" The time between each message is longer than the time between normal conversations (for example, 5 seconds apart). Here, the vertical movement pattern detected by the accelerometer 311 strongly suggests that the user is experiencing greater vertical movement intensity than during normal walking, i.e., running, jumping, or bouncing on playground equipment such as a trampoline. The information processing device 100 analyzes this behavior and estimates that the fragmented utterances, such as "Right now," "I'm jumping," and "It's fun!", are occurring within the same context, and integrates them as a continuous context.

[0145] The second terminal 300 transmits sensor data indicating vigorous vertical movement detected by the accelerometer 311 to the information processing device 100 along with each voice message. The message group identification unit 131 of the information processing device 100 would normally determine that the messages are unrelated due to the long time interval, but instead, it determines from the accelerometer data that the user is "engaged in vigorous exercise".

[0146] Based on this determination, the time threshold for identifying the message group is dynamically extended from the usual 3 seconds to 10 seconds. As a result, the three voice messages are correctly identified as a "series of utterances" and linked together by the integrated processing unit 132.

[0147] The conversion unit 130 generates a single text message, "Right now, I'm bouncing around, it's fun!", and sends it to the first terminal 200. This allows the parent to accurately understand the child's situation.

[0148] (Example 4) This embodiment assumes a scenario in which a second user P2 (child) reports their arrival at school. The parent (first user P1) has previously set the grounds of the school the child attends as the "school" geofence using the geofence setting unit 240 of the first terminal 200.

[0149] Immediately after passing through the school gate, the child presses a button on the second terminal 300 and sends a single message saying, "I'm here." The information processing device 100 receives this voice message and location information sent from the GPS receiver 309. The geofence determination unit 160 determines that the location information is within the range of the "school" geofence.

[0150] The value-added processing unit 134 adds contextual information to the generated text "I'm here!" based on this geofence information. Specifically, it completes the text to "I've arrived at school!" and automatically adds a schoolbag emoji.

[0151] Since this processing result is sent to the first terminal 200, parents can understand more clearly and in a heartwarming way that their child has arrived at school safely just by taking a look at the message.

[0152] (Example 5) This embodiment demonstrates the operation of Modification 1 (client-centric configuration). Second user P2 (child) sends multiple voice messages ("Fish", "I caught one", "It's big") from a campsite in a mountainous area. At that time, the first terminal 200 of the first user P1 (guardian) was out of range (offline).

[0153] The information processing device 100 holds the messages in a queue without converting them and waits for the first terminal 200 to come back online. When the parent returns to town and the first terminal 200 reconnects to the network, it receives three voice message data relayed from the information processing device 100.

[0154] The conversion unit operating on the processor 201 of the first terminal 200 concatenates the three received audio data, performs speech recognition processing, generates a single text message, "I caught a fish, it's big," and displays it on the display 205. In this process, the audio data is not stored on the server, thus protecting privacy.

[0155] (Example 6) This embodiment assumes a scenario in which a second user P2 (a child) intermittently reports on things that interest them. The child first sends a voice message saying, "Oh, a fire truck." Twenty seconds later, they send a second message saying, "It's big and red."

[0156] The information processing device 100 would determine that the two messages are separate utterances based solely on the temporal condition, since the interval between receiving the two messages is 20 seconds. However, the message group identification unit 131 in this embodiment performs a unification determination based on semantic relevance. First, each audio data is tentatively converted into text to obtain two provisional texts: "fire truck" and "big and red".

[0157] Next, the relationship between these texts is analyzed. The adjectives "big" and "red" match the typical characteristics of a vehicle, particularly a "fire truck," and it is determined that there is a very high semantic relationship. Based on this determination, the message group identification unit 131 overrides the temporal conditions and identifies these two audio messages as a "series of utterances" that should be merged.

[0158] Finally, the conversion unit 130 concatenates the two audio data to generate a single, contextually coherent text message, "Oh, it's a fire truck. It's big and red," and sends it to the first terminal 200.

[0159] It goes without saying that the present invention is not limited to the embodiments described above, and various modifications, improvements, and combinations are possible without departing from the spirit of the invention. For example, the components described in each embodiment may be combined in appropriate ways. [Explanation of Symbols]

[0160] 1. Asymmetric communication system 100... Information processing equipment (server) 101, 201, 301… Processors 102, 202, 302...Main storage 103, 203, 303…Auxiliary storage device 104, 204, 304… Communication Interfaces 110, 210, 310… Control Units 120, 220... Receiver 130...Conversion section 131...Message group identification unit 132...Integrated Processing Unit 133... Voice recognition unit 134...Value-added processing unit 140...Transmitter 150...Location information management department 160...Geofence determination unit 170... Notification Control Unit 200...First terminal 205…Display 206…Input devices 230...Display Control Unit 240...Geofence Setting Section 300...Second terminal 305…Display 306…Input devices 307... Mike 308...Speaker 309…GPS receiver 311...Accelerometer 331... Send queue 332...Presence buffer 400…Display screen 401... speech bubble 402…Text message 403... Map thumbnail 404…Location information text 405...Integrated Mark 406... Candidate display area NW...Network P1...First User P2...Second User

Claims

1. An information processing device capable of communicating with a first terminal and a second terminal, A receiving unit that receives multiple voice message data transmitted from the second terminal, A conversion unit generates a single text data based on the plurality of voice message data received by the receiving unit, A transmission unit that transmits the single text data generated by the conversion unit to the first terminal, An information processing device equipped with the following features.

2. The information processing apparatus according to claim 1, wherein the plurality of voice message data are transmitted from the second terminal which does not have a user interface capable of text input.

3. The information processing apparatus according to claim 2, wherein the plurality of voice message data are transmitted from the second terminal which has a total of three or fewer physical operators.

4. The information processing apparatus according to claim 1, wherein the conversion unit generates the single text data based on the plurality of voice message data received within a predetermined time.

5. The information processing apparatus according to claim 1, wherein the conversion unit generates the single text data while maintaining the reception order of the plurality of voice message data.

6. The information processing apparatus according to claim 1, wherein the transmitting unit transmits location information received from the second terminal, which is temporally related to the plurality of voice message data, along with the single text data.

7. The information processing apparatus according to claim 1, wherein the transmitting unit transmits text data corresponding to the first received voice message data from among the plurality of voice message data as a preliminary report to the first terminal, and thereafter transmits the single text data to the first terminal as a final version.

8. The information processing apparatus according to claim 1, wherein the conversion unit further performs summarization processing on the text generated by speech recognition processing and generates summarized text as a single text data.

9. The information processing apparatus according to claim 1, wherein the transmitting unit transmits identification information indicating that the single text data was generated based on the plurality of voice message data to the first terminal together with the single text data.

10. The conversion unit generates multiple candidate text data if the confidence level of the speech recognition processing result falls below a predetermined threshold. The information processing apparatus according to claim 1, wherein the transmitting unit transmits the plurality of candidate text data to the first terminal.

11. The information processing apparatus according to claim 1, wherein the conversion unit generates the single text data using a personalized language model based on the communication history of a specific user group.

12. The receiving unit further receives sensor data indicating the activity status of the user of the second terminal, which is acquired from the sensor mounted on the second terminal. The information processing apparatus according to claim 1, wherein the conversion unit determines whether or not to generate the single text data based on the plurality of voice message data based on the sensor data.

13. The information processing apparatus according to claim 1, wherein the conversion unit adds additional information to the single text data based on information regarding a specific geographical range defined based on the geographical location of the second terminal.

14. The conversion unit analyzes the acoustic characteristics of the voice message data to generate information indicating the emotions or health status of the user of the second terminal. The information processing apparatus according to claim 1, wherein the transmitting unit transmits the generated information together with the single text data to the first terminal.

15. An information processing method performed by the processor of an information processing device capable of communicating with a first terminal and a second terminal, The steps include receiving multiple voice message data transmitted from the second terminal, The steps include generating a single text data based on the multiple voice message data received, The steps include sending the generated single text data to the first terminal, An information processing method having

16. The computer of the information processing device that can communicate with the first terminal and the second terminal, The terminal receives multiple voice message data transmitted from the second terminal. Based on the multiple voice message data received, a single text data is generated. The single text data generated above is sent to the first terminal. A program that executes a process.

17. A second terminal capable of communicating with an information processing device, An input receiving unit that receives instructions to record voice messages, A storage unit equipped with a transmission queue for storing recorded voice message data, A communication unit that transmits the voice message data stored in the transmission queue to the information processing device, It comprises a display control unit that displays the status of the message on the display unit, The display control unit selects one display mode from a predetermined set of different display modes according to the state of the transmission queue and the state of the communication unit, and displays it on the display unit. Terminal 2.

18. The aforementioned input receiving unit has a recording button equipped with a touch sensor. The second terminal according to claim 17, wherein the control unit starts pre-recording audio from the moment it detects contact with the touch sensor, and stores the audio from a predetermined time prior to the moment it detects the pressing of the recording button in the transmission queue.

19. The second terminal according to claim 18, wherein the control unit, after detecting the end of pressing the recording button, continues recording the voice for a predetermined time if contact with the touch sensor continues.

20. A first terminal capable of communicating with an information processing device, A receiving unit receives a single text data, which includes information indicating that it was generated based on multiple voice message data, and location information from the aforementioned information processing device. A display control unit that displays the single text data and the location information received by the receiving unit on the display unit in relation to each other, A first terminal equipped with [the following].

21. The receiving unit further receives identification information indicating that the single text data was generated based on multiple voice message data, The first terminal according to claim 20, wherein the display control unit displays, based on the identification information, that the single text data is an integrated message.

22. The receiving unit further receives multiple candidate text data, The first terminal according to claim 20, wherein the display control unit causes the plurality of candidate text data to be displayed on the display unit.

23. The system comprises a server, a first terminal, and a second terminal. The aforementioned server, A receiving unit that receives multiple voice message data transmitted from the second terminal, A conversion unit generates a single text data based on the plurality of voice message data received by the receiving unit, A transmission unit that transmits the single text data generated by the conversion unit to the first terminal, An asymmetrical communication system equipped with [features / equipment].

Images