Data processing device, data processing system, and audio processing method

Abstract

To allow data arrival status to be confirmed without confirmation by conversation between users.SOLUTION: A data processing device includes: a receiving unit that receives first coded data from another data processing device; a decoding unit that decodes the first coded data received by the receiving unit; encoding unit that encodes data obtained by the decoding unit to generate second coded data; and a control unit that controls transmission of the second coded data to the other communication device when it is determined that the first coded data is data transmitted in a specific operation mode from the other communication device.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a data processing device, a data processing system, and an audio processing method. [Background technology]

[0002] In recent years, companies have been accelerating their introduction of teleworking as a measure against the pandemic. In teleworking, teleworkers in the office and at home, or teleworkers at home, are connected via the Internet, and remote meetings, training, and remote communication are conducted using network-compatible TV conferencing systems or video calling software on PCs.

[0003] In the above TV conference system and video call software, depending on the type and configuration of the device used, problems may occur with the video and audio when participating in a conference. For this reason, a mechanism is known that allows a user to check the operation of a TV conference system before participating in a conference. For example, Patent Document 1 discloses a technology that has a normal mode used in a conference and a self-check mode for advance check, and allows the quality of the audio transmitted to the other party during a remote conference to be checked. In this technology, in the self-check mode, the user's voice is compressed, and then the compressed voice is looped back in the user's terminal, and the voice is expanded and then played back. This allows the user to check the degree of degradation caused by the compression and expansion of the audio. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2011-193374 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, even if a user's voice data is transmitted to the terminals of other users participating in the conference, the user's voice may not be output from the other users' terminals due to problems with the network or the other users' terminals. For this reason, each time a conference is held, participating users often ask each other by conversation, "Has your voice been received?" This is repeated every time a new user is added, and conversely, when a user joins a conference midway, the conference cannot be interrupted, and confirmation may not be possible. Note that the technology described in Patent Document 1 is a technology that loops back the user's voice within the user's terminal, so with the technology described in Patent Document 1, it is difficult to confirm whether the user's voice data is received by the terminals of other users participating in the conference.

[0006] Therefore, the present invention has been made in consideration of the above problems, and an object of the present invention is to provide a new and improved data processing device, data processing system, and voice processing method that make it possible to check the delivery status of data without confirmation by conversation between users. [Means for solving the problem]

[0007] In order to solve the above problem, according to one aspect of the present invention, there is provided a data processing apparatus comprising: a first encoding unit that encodes input data to generate encoded data; a receiving unit that receives the first encoded data from another data processing apparatus; a decoding unit that decodes the first encoded data received by the receiving unit; and a second encoding unit that encodes data obtained by the decoding unit to generate second encoded data. Controlling the encoded data generated by the first encoding unit so that it is transmitted; When it is determined that the first encoded data is data transmitted from the other data processing device in a specific operation mode, 、 Controlling the second encoded data to be transmitted to the other data processing device. This can be done in parallel. A data processing device is provided, comprising:

[0008] The data processing device further includes an output unit that outputs audio or video based on the data obtained by the decoding unit, and the output unit may not output audio and video based on the data obtained by the decoding unit when the control unit determines that the first encoded data is data transmitted from the other data processing device in the specific operating mode.

[0009] The above Second The encoding section may generate the second encoded data using a second processing method that causes less degradation in quality than a first processing method used to generate the first encoded data.

[0010] The control unit may control transmission of the second encoded data using a second communication method that is more reliable than a first communication method used for communicating the first encoded data.

[0011] The control unit may determine, based on a predetermined flag being added to the first encoded data, that the first encoded data is data transmitted from the other data processing device in a specific operating mode.

[0012] In order to solve the above problem, according to another aspect of the present invention, there is provided a data processing system having a first data processing device and a second data processing device, wherein the first data processing device transmits first coded data obtained by coding data to the second data processing device, and the second data processing device includes a first coding unit that codes input data to generate coded data, a receiving unit that receives the first coded data from the first data processing device, a decoding unit that decodes the first coded data received by the receiving unit, and a second coding unit that codes data obtained by the decoding unit to generate second coded data. Controlling the encoded data generated by the first encoding unit so that it is transmitted; When it is determined that the first encoded data is data transmitted from the first data processing device in a specific operation mode, the second encoded data is controlled to be transmitted to the first data processing device. This can be done in parallel.A data processing system is provided, comprising: a controller.

[0013] In order to solve the above problem, according to another aspect of the present invention, a method for encoding input data by a first encoding unit to generate encoded data, receiving the first encoded data from another data processing device, decoding the first encoded data, and encoding data obtained by decoding the first encoded data by a second encoding unit to generate second encoded data; Controlling the encoded data generated by the first encoding unit so that it is transmitted; When it is determined that the first encoded data is data transmitted from the other data processing device in a specific operation mode, 、 Controlling the second encoded data to be transmitted to the other data processing device. To execute these things in parallel A method for processing audio is provided, comprising: Effect of the Invention

[0021] According to the present invention as described above, it is possible to check the data arrival status without confirmation through conversation between users. [Brief description of the drawings]

[0022] [Figure 1] 1 is an explanatory diagram showing the configuration of a data processing system according to an embodiment of the present invention; [Diagram 2] FIG. 1 is an explanatory diagram showing a configuration of a voice processing device 20 according to an embodiment of the present invention. [Diagram 3] FIG. 2 is an explanatory diagram showing a connection sequence in a data processing system according to an embodiment of the present invention. [Figure 4] FIG. 11 is an explanatory diagram showing a processing sequence when the voice processing device 20A transitions to a test mode. [Diagram 5] 4 is an explanatory diagram showing the flow of audio data in a normal mode of the audio processing device 20A and the audio processing device 20B. FIG. [Figure 6]FIG. 11 is an explanatory diagram showing the flow of audio data when the audio processing device 20 operates in a test mode and the audio processing device 20B operates in a normal mode. [Figure 7] 2 is a block diagram showing a hardware configuration of a voice processing device 20. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and duplicated explanations will be omitted.

[0024] In addition, in this specification and drawings, multiple components having substantially the same functional configuration may be distinguished by adding different alphabets after the same reference numeral. For example, multiple components having substantially the same functional configuration or logical meaning are distinguished as necessary, such as audio processing devices 20A, 20B, and 20C. However, when there is no need to particularly distinguish between multiple components having substantially the same functional configuration, only the same reference numerals are attached to each of the multiple components. For example, when there is no need to particularly distinguish between audio processing devices 20A, 20B, and 20C, each audio processing device is simply referred to as audio processing device 20.

[0025] <1. Overview of the data processing system> An embodiment of the present invention relates to a data processing system for realizing an audio conference in which a plurality of users from remote locations participate. First, an overview of the data processing system according to an embodiment of the present invention will be described with reference to FIG.

[0026] 1 is an explanatory diagram showing the configuration of a data processing system according to an embodiment of the present invention. As shown in FIG. 1, the data processing system according to an embodiment of the present invention includes voice processing devices 20A to 20F and a conference server 30.

[0027] These voice processing devices 20A to 20F and the conference server 30 are connected via a network 12. The network 12 is a wired or wireless transmission path for information transmitted from devices connected to the network 12. For example, the network 12 may include public line networks such as the Internet, a telephone line network, and a satellite communication network, as well as various LANs (Local Area Networks) including Ethernet (registered trademark), and WANs (Wide Area Networks). The network 12 may also include a dedicated line network such as an IP-VPN (Internet Protocol-Virtual Private Network).

[0028] In the example shown in FIG. 1, the voice processing device 20A, the voice processing device 20B, and the voice processing device 20C belong to a group G1 in which they participate in the same voice conference, the voice processing device 20D and the voice processing device 20E belong to a group G2 in which they participate in another voice conference, and the voice processing device 20F does not participate in any voice conference.

[0029] 1, user UA uses voice processing device 20A, user UB uses voice processing device 20B, user UC uses voice processing device 20C, user UD uses voice processing device 20D, user UE uses voice processing device 20E, and user UF uses voice processing device 20F. However, the number of voice processing devices 20 constituting the data processing system and the number of users U using the data processing system may be less or more.

[0030] (Conference server) The conference server 30 manages the participation and withdrawal of each audio processing device 20 in a conference. For example, the conference server 30 manages the participation and withdrawal of each audio processing device 20 in a conference by using a conference protocol such as WebRTC (Web Real-Time Communication). Communication of video data and audio data is performed within a group such as the above-mentioned group G1 and group G2 without going through the conference server 30. Note that when the audio processing devices 20 are connected to each other via Peer to Peer, the conference server 30 does not need to be provided.

[0031] (Audio processing device) The voice processing device 20 is an example of a data processing device, and transmits voice data indicating a voice uttered by a user of the voice processing device 20 to the other voice processing device 20. In addition, the voice processing device 20 receives voice data indicating a voice uttered by a user of the other voice processing device 20 from the other voice processing device 20, and outputs the voice uttered by the user of the other voice processing device 20 based on the voice data.

[0032] For example, in the example shown in Fig. 1, the voice processing device 20A transmits voice data indicating the voice uttered by the user UA to the voice processing device 20B and the voice processing device 20C, and the voice processing device 20B and the voice processing device 20C output the voice uttered by the user UA based on the voice data. In addition, the voice processing device 20B transmits voice data indicating the voice uttered by the user UB to the voice processing device 20A, and the voice processing device 20A outputs the voice uttered by the user UB based on the voice data. In addition, the voice processing device 20C transmits voice data indicating the voice uttered by the user UC to the voice processing device 20A, and the voice processing device 20A outputs the voice uttered by the user UC based on the voice data. With this configuration, the users UA, UB, and UC can hold an audio conference.

[0033] In addition to audio data, the audio processing device 20 may transmit and receive video data to and from other audio processing devices 20. In addition, while a notebook PC (Personal Computer) is shown as an example of the audio processing device 20 in Fig. 1, the audio processing device 20 may be other information processing devices such as a desktop PC, a smartphone, a mobile phone, or a PHS (Personal Handyphone System).

[0034] (background) In such a data processing system, even if a user's voice data is transmitted to the voice processing device of another user participating in the conference, the user's voice may not be output from the other user's voice processing device due to problems with the network or the voice processing device of the other user. For this reason, each time a conference is held, participating users often ask each other by conversation, "Is your voice being received?" This is repeated every time a user adds a user, and conversely, if a user joins a conference midway, the conference cannot be interrupted, and confirmation may not be possible.

[0035] In some cases, a problem with the output of the receiving voice processing device may be pointed out as if it were a problem with the sending voice processing device, and time may be wasted in resolving the problem. Conversely, a user may mistakenly believe that a problem with the receiving voice processing device is a problem with the sending voice processing device. Furthermore, remote conferences are highly dependent on the network environment, and packet delays and losses can cause a decrease in the quality of the voice data, but it is difficult to check the operation during a conference.

[0036] It is also possible to configure the conference server so that the video data and audio data can be looped back on the conference server side to check the operation of the audio processing device. However, in a system in which the video data and audio data of each audio processing device are sent and received during a conference without going through the conference server as shown in Fig. 1, it is difficult to check the arrival status of audio data from one audio processing device to another during the conference.

[0037] The inventor of the present invention has come to create an embodiment of the present invention by focusing on the above-mentioned circumstances. According to an embodiment of the present invention, it is possible to check the arrival status of voice data without confirmation by conversation between users. As a result, it is possible to smoothly conduct an audio conference. The configuration and operation of such an embodiment of the present invention will be described in detail below.

[0038] <2. Overview of voice processing device> Fig. 2 is an explanatory diagram showing a configuration of a voice processing device 20 according to an embodiment of the present invention. As shown in Fig. 2, the voice processing device 20 according to an embodiment of the present invention includes a voice input unit 220, a first encoding unit 224, a first communication unit 228, a first decoding unit 232, a voice output unit 236, an operation unit 238, a control unit 240, a first buffer 244, a forwarding unit 248, a second encoding unit 252, a second communication unit 256, a second decoding unit 260, and a second buffer 264.

[0039] (Voice input unit 220) The voice input unit 220 is configured to receive input of a voice uttered by a user of the voice processing device 20. The voice input unit 220 converts the voice uttered by the user of the voice processing device 20 into electrical voice data, and supplies the voice data to the first encoding unit 224 and the first buffer 244. The voice input unit 220 may be configured of a microphone device, or may be a microphone mounted on a camera device used in a conference.

[0040] (First encoding unit 224) The first encoding unit 224 generates encoded audio data by encoding the audio data supplied from the audio input unit 220 using a first processing method. The first processing method may be a lossy compression method. In this specification, the encoded audio data generated by the first encoding unit 224 may be referred to as first encoded audio data.

[0041] (First communication unit 228) The first communication unit 228 functions as a transmitting unit (first transmitting unit) that transmits the encoded audio data generated by the first encoding unit 224 to another audio processing device 20, and as a receiving unit (first receiving unit) that receives the encoded audio data generated by the first processing method from the other audio processing device 20. The first communication unit 228 communicates using a conference protocol such as UDP as the first communication method. The first communication unit 228 may use TCP as another protocol, but it is expected that when packet delay becomes large, packets will be discarded to eliminate the delay.

[0042] (First decoding unit 232) The first decoding unit 232 decodes the encoded audio data received from the other audio processing device 20 by the first communication unit 228 using the first processing method.

[0043] (Audio output unit 236) The audio output unit 236 outputs the audio uttered by the user of the other audio processing device 20 based on the audio data obtained by the first decoding unit 232. The audio output unit 236 may be a speaker device, a headset integrated with a microphone device, a microphone speaker, or the like. The audio output unit 236 also outputs audio supplied from a second buffer 264 described later.

[0044] The operation modes of the voice processing device 20 include a normal mode, which is an example of a first operation mode, and a test mode, which is an example of a second operation mode (specific operation mode). In the normal mode, the above-mentioned voice input unit 220, the first encoding unit 224, the first communication unit 228, the first decoding unit 232, and the voice output unit 236 operate, so that it is possible to hold an audio conference between users of the voice processing device 20. In the test mode, it is possible to check whether or not the voice data of the user of the voice processing device 20 has reached another voice processing device 20, and if so, what quality the voice data has reached, through the operation of the second communication unit 256, the second decoding unit 260, the second buffer 264, and the like, which will be described later.

[0045] (Operation unit 238) The operation unit 238 is configured to be operated by a user of the voice processing device 20 to input information, instructions, or the like to the voice processing device 20. For example, the user of the voice processing device 20 inputs, to the operation unit 238, an instruction to switch the operation mode of the voice processing device 20 between a normal mode and a test mode.

[0046] (Control unit 240) The control unit 240 controls the overall operation of the voice processing device 20. For example, the control unit 240 switches the operation mode of the voice processing device 20 between a normal mode and a test mode in accordance with a user's instruction to the operation unit 238. In the test mode, the control unit 240 performs, for example, the following control.

[0047] The control unit 240 stops the supply of audio data from the audio input unit 220 to the first encoding unit 224. Instead, the control unit 240 supplies the audio data held in the first buffer 244 to the first encoding unit 224. The first encoding unit 224 generates encoded audio data by encoding the audio data supplied from the first buffer 244. In accordance with an instruction from a user, the audio data supplied from the audio input unit 220 to the first encoding unit 224 may be encoded without using the first buffer 244 to generate encoded audio data.

[0048] The control unit 240 causes the first communication unit 228 to transmit information about the test mode together with the encoded audio data. The information about the test mode includes a predetermined flag, and information indicating an address and a port for waiting for data from another audio processing device 20. When the first communication unit 228 performs communication using WebRTC, the first communication unit 228 may transmit the information about the test mode using, for example, a data channel.

[0049] The control unit 240 stops at least one of the following: reception of encoded audio data by the first communication unit 228, supply of encoded audio data from the first communication unit 228 to the first decoding unit 232, decoding of encoded audio data by the first decoding unit 232, or supply of audio data from the first decoding unit 232 to the audio output unit 236.

[0050] With the second decoding unit 260 and the second buffer 264 in an operating state, the second decoding unit 260 decodes encoded audio data (second encoded audio data generated by a second processing method) received from another audio processing device 20 by the second communication unit 256, the audio data obtained by decoding is stored in the second buffer 264, and one or more pieces of audio data stored in the second buffer 264 are sequentially output to the audio output unit 236.

[0051] In the normal mode, the control unit 240 performs the following control, for example.

[0052] The control unit 240 puts the second decoding unit 260 and the second buffer 264 in an inoperative state.

[0053] When the first communication unit 228 receives information indicating a test mode from another audio processing device 20 together with encoded audio data, the control unit 240 determines that the encoded audio data is data transmitted in test mode from the other audio processing device 20.

[0054] If the encoded audio data is data transmitted in the test mode from another audio processing device 20, the control unit 240 does not supply the audio data obtained from the encoded audio data by the first decoding unit 232 to the audio output unit 236. Instead, the control unit 240 causes the forwarding unit 248 to forward the audio data to the second encoding unit 252. As a result, the second encoding unit 252 encodes the audio data using the second processing method to generate encoded audio data, and the second communication unit 256 transmits the encoded audio data to the other audio processing device 20 using the second communication method.

[0055] (First buffer 244) The first buffer 244 temporarily holds a part of the voice data supplied from the voice input unit 220 as test voice data. In the test mode, the voice data is supplied from the first buffer 244 to the first encoding unit 224 based on an operation of the operation unit 238 by the user.

[0056] (Transportation Section 248) When the first communication unit 228 receives information indicating a test mode together with encoded audio data from another audio processing device 20, the forwarding unit 248 transfers the audio data obtained by decoding the encoded audio data from the first decoding unit 232 to the second encoding unit 252.

[0057] (Second encoding unit 252) The second encoding unit 252 generates encoded audio data by encoding the audio data received from the forwarding unit 248 using a second processing method. The second processing method is a processing method that causes less degradation of audio quality than the first processing method. The second processing method may be a lossless encoding method that does not cause degradation of audio data. In this specification, the encoded audio data generated by the second encoding unit 252 may also be referred to as second encoded audio data.

[0058] (Second communication unit 256) The second communication unit 256 functions as a transmission unit (second transmission unit) that transmits the encoded voice data generated by the second encoding unit 252 together with information indicating the test mode to the other voice processing device 20 that transmitted the encoded voice data, and as a reception unit (second reception unit) that receives the encoded voice data generated by the second processing method from the other voice processing device 20. The second communication unit 256 may perform communication using a second communication method that is more reliable than the first communication method. An example of such a second communication method is TCP, which prioritizes the preservation of voice data even if a packet delay occurs. Note that the second communication unit 256 and the first communication unit 228 may be functions realized in the same network device.

[0059] (Second Decoding Unit 260) The second decoding unit 260 decodes the encoded audio data received from the other audio processing device 20 by the second communication unit 256 using the second processing method.

[0060] (Second buffer 264) The second buffer 264 temporarily holds the audio data obtained by the second decoding unit 260. When there are a plurality of other audio processing devices 20 communicating with the audio processing device 20 and the encoded audio data is received by the second communication unit 256 from the plurality of audio processing devices 20, the second buffer 264 holds a plurality of audio data obtained by decoding the plurality of encoded audio data. The plurality of audio data held by the second buffer 264 is sequentially output as audio from the audio output unit 236 under the control of the control unit 240. Note that when there is only one other audio processing device 20 communicating with the audio processing device 20, the second buffer 264 does not hold audio data, and the audio output unit 236 may output audio based on the audio data.

[0061] <3.Operation> The configuration of the voice processing device 20 according to an embodiment of the present invention has been described above. Next, the operation of the data processing system according to an embodiment of the present invention will be described.

[0062] (Connection sequence) First, with reference to FIG. 3, a connection sequence in a data processing system according to an embodiment of the present invention will be described.

[0063] Fig. 3 is an explanatory diagram showing a connection sequence in a data processing system according to an embodiment of the present invention. As shown in Fig. 3, first, the user UA performs an operation to connect to a conference on the operation unit 238 of the voice processing device 20A (S1). The connection to the conference may be made by specifying a conference room prepared in advance by the conference server 30, or by creating a new conference room. The voice processing device 20A transmits a connection request to the conference server 30 according to the operation from the user UA (S2).

[0064] Similarly, when the user UB first performs a connection operation to a conference on the operation unit 238 of the voice processing device 20B (S3), the voice processing device 20B transmits a connection request to the conference server 30 (S4). When the voice processing device 20A and the voice processing device 20B specify the same conference room, the conference server 30 issues a connection instruction to the voice processing device 20A and the voice processing device 20B to communicate with each other in a Peer-to-Peer manner (S5, S6), and the voice processing device 20A and the voice processing device 20B establish a connection (S7, S8).

[0065] After the connection is established, when the user UA speaks to the voice processing device 20A (S9), the first encoding unit 224 of the voice processing device 20A generates encoded voice data and transmits the encoded voice data to the voice processing device 20B (S10). Then, the first decoding unit 232 of the voice processing device 20B decodes the encoded voice data, and the voice output unit 236 of the voice processing device 20B outputs the voice of the user UA based on the voice data obtained by decoding (S11).

[0066] Similarly, when the user UB speaks to the voice processing device 20B (S12), the first encoding unit 224 of the voice processing device 20B generates encoded voice data and transmits the encoded voice data to the voice processing device 20A (S13). Then, the first decoding unit 232 of the voice processing device 20A decodes the encoded voice data, and the voice output unit 236 of the voice processing device 20A outputs the voice of the user UB based on the voice data obtained by decoding (S14). During this time, the encoded voice data does not pass through the conference server 30. Even if the number of voice processing devices 20 participating in the conference room increases, each voice processing device 20 can be connected using a similar sequence.

[0067] (Test mode sequence) Next, a processing sequence will be described in the case where the voice processing device 20A transitions to a test mode after the connection between the voice processing device 20A and the voice processing device 20B is established.

[0068] 4 is an explanatory diagram showing a processing sequence when the voice processing device 20A transitions to a test mode. When the user UA performs an operation to start a voice test on the voice processing device 20A (S21), the voice processing device 20A transitions to a test mode (S22). In the test mode, when transmitting encoded voice data (described later), a flag is set to the other device to indicate that the voice data is for testing.

[0069] Thereafter, the user UA issues an instruction as to whether to make an utterance in the test mode or to use the voice data stored in the first buffer 244 (S23). When the user UA makes a test utterance (S24), the first buffer 244 of the voice processing device 20A stores the voice data for a certain period of time (S25: first buffering), and the first encoding unit 224 encodes the stored voice data using a first processing method to generate encoded voice data (S27: first encoding).

[0070] On the other hand, when using audio data stored in the first buffer 244, if there is audio data stored in the first buffer 244, the control unit 240 supplies the audio data to the first encoding unit 224 (S26), and the first encoding unit 224 encodes the audio data using a first processing method to generate encoded audio data (S27: first encoding).

[0071] In either case, the encoded audio data is transmitted from the first communication unit 228 of the audio processing device 20A to the audio processing device 20B together with information on the test mode including a predetermined flag (S28). The information on the test mode includes, in addition to the predetermined flag, information indicating an address and port for waiting for data from the audio processing device 20B, etc.

[0072] When the first communication unit 228 of the voice processing device 20B receives the encoded voice data, the first decoding unit 232 of the voice processing device 20B decodes the encoded voice data using a first processing method (S29: first decoding). Here, when the control unit 240 of the voice processing device 20B determines that the encoded voice data is data transmitted in a test mode based on the setting of a predetermined flag, the forwarding unit 248 of the voice processing device 20B forwards the voice data obtained by decoding to the second encoding unit 252, and the second encoding unit 252 encodes the voice data using a second processing method (S30: second encoding). Then, the second communication unit 256 of the voice processing device 20B transmits the encoded voice data generated by encoding to the voice processing device 20A (S31).

[0073] When the second communication unit 256 of the audio processing device 20A receives the encoded audio data, the second decoding unit 260 of the audio processing device 20A decodes the encoded audio data using a second processing method (S32: second decoding), and the second buffer 264 temporarily holds the audio data obtained by decoding (S33: second buffering).

[0074] Thereafter, the audio output unit 236 outputs audio based on the audio data stored in the second buffer 264 (S34). If there are multiple devices that are the counterparts of the audio processing device 20A, the audio output unit 236 plays back the multiple pieces of audio data stored in the second buffer 264 in any order. Then, the control unit 240 performs processing to end the test mode, and clears the setting of a predetermined flag (S35).

[0075] (Audio data flow) Here, the flow of audio data in the normal mode and the flow of audio data in the test mode will be summarized with reference to FIG. 5 and FIG.

[0076] Fig. 5 is an explanatory diagram showing the flow of voice data in normal mode of the voice processing device 20A and the voice processing device 20B. In Fig. 5, a solid line indicates the flow of voice data of a user UA of the voice processing device 20A, and a two-dot chain line indicates the flow of voice data of a user UB of the voice processing device 20B.

[0077] As shown by the solid line in Fig. 5, in the normal mode, the voice data of the user UA is processed in the following order: voice input unit 220, first encoding unit 224 (may be via first buffer 244), first communication unit 228 of the voice processing device 20A, and then the first communication unit 228, first decoding unit 232, and voice output unit 236 of the voice processing device 20B. Similarly, as shown by the two-dot chain line in Fig. 5, the voice data of the user UB is processed in the following order: voice input unit 220, first encoding unit 224 (may be via first buffer 244), first communication unit 228 of the voice processing device 20B, and then the first communication unit 228, first decoding unit 232, and voice output unit 236 of the voice processing device 20A.

[0078] Fig. 6 is an explanatory diagram showing the flow of voice data when the voice processing device 20 operates in the test mode and the voice processing device 20B operates in the normal mode. In Fig. 6, the solid line indicates the flow of voice data of the user UA of the voice processing device 20A, and the two-dot chain line indicates the flow of voice data of the user UB of the voice processing device 20B.

[0079] 6, in the test mode, the voice data of the user UA is processed in the flow of the voice input unit 220, the first encoding unit 224 (may be via the first buffer 244), the first communication unit 228 of the voice processing device 20A, and then in the flow of the first communication unit 228, the first decoding unit 232, the forwarding unit 248, the second encoding unit 252, and the second communication unit 256 of the voice processing device 20B, and the encoded voice data generated by the second processing method is transmitted to the voice processing device 20A. After that, the encoded voice data is processed in the flow of the second communication unit 256, the second decoding unit 260, the second buffer 264, and the voice output unit 236 of the voice processing device 20A, and the voice of the user UA is output from the voice output unit 236 of the voice processing device 20A. In addition, in the voice processing device 20B, since no voice data is supplied from the first decoding unit 232 to the voice output unit 236, no voice data of the user UA is output from the voice output unit 236 of the voice processing device 20B.

[0080] 6, when the voice processing device 20A is operating in the test mode, the voice data of the user UB is processed in the following order: voice input unit 220 of the voice processing device 20B, first encoding unit 224 (which may be via the first buffer 244), first communication unit 228, and then first communication unit 228 and first decoding unit 232 of the voice processing device 20A. Since the voice data obtained by the first decoding unit 232 is not supplied to the voice output unit 236, the voice data of the user UB is not output from the voice output unit 236 of the voice processing device 20A.

[0081] <4. Effects> According to the embodiment of the present invention described above, various operational effects are achieved. For example, in a test mode, the voice processing device 20 according to the embodiment of the present invention receives a user's voice data from another voice processing device 20 and outputs the user's voice based on the received voice data. Therefore, it is possible to check whether the environment allows the user's voice data to reach another voice processing device 20 without having to check by conversation between users such as "Is your voice reaching you?" As a result, the progress of the conference is not hindered by such conversation, and the conference can proceed smoothly.

[0082] Furthermore, the voice processing device 20 according to an embodiment of the present invention supports a lossless coding method (second processing method) and a reliable transmission and reception protocol (second communication method) in addition to the coding method (first processing method) and the speed-oriented transmission and reception protocol (first communication method) used in normal conferences and conversations in a network-compatible TV conference system or video call software on a PC. The second processing method and the second communication method are used to return voice data from the other voice processing device 20. Therefore, the user of the voice processing device 20 can check the quality of the voice that will be output by the other voice processing device 20 based on his or her own voice output from the voice processing device 20. Furthermore, if there is no problem with the quality of the user's own voice output from the voice processing device 20, it is known that there is no problem with the internal processing of the voice processing device 20 and the other voice processing device 20 and the networks of both, so that the time lost in investigating the cause of the trouble can be reduced.

[0083] Moreover, in the test mode, the voice processing device 20 according to the embodiment of the present invention does not output voice based on normal voice data received from another voice processing device 20. Therefore, the user of the voice processing device 20 can more accurately grasp the quality of the voice by clearly listening to his / her own voice received by the other voice processing device 20. Moreover, since the other voice processing device 20 does not output voice based on voice data transmitted from the voice processing device 20 operating in the test mode, it is possible to prevent the conference from being disturbed by the test voice.

[0084] In addition, since the voice processing device 20 according to one embodiment of the present invention is equipped with the second buffer 264, even when the voice processing device 20 is in a conference with multiple other voice processing devices 20, it is possible to listen to its own voice received back from each of the other voice processing devices 20 in turn.

[0085] <5. Modifications> An embodiment of the present invention has been described above. In the following, some modified examples of the above-described embodiment will be described. Note that each modified example described below may be applied alone to the above-described embodiment, or may be applied in combination to the above-described embodiment. Also, each modified example may be applied in place of the configuration of the above-described embodiment, or may be applied in addition to the configuration of the above-described embodiment.

[0086] In the above, the second processing method is described as being different from the encoding method (first processing method) used in normal conferences and conversations. However, the type of the second processing method may be the same as the type of the first processing method, and the parameters applied to the second processing method may be different from the parameters applied to the first processing method. In addition, the second processing method may not be a completely lossless method, but may be near-lossless.

[0087] Also, in the above example, the voice processing device 20 supports a lossless coding method (second processing method) and a reliable transmission and reception protocol (second communication method) in addition to the coding method (first processing method) and the speed-oriented transmission and reception protocol (first communication method) for normal conferences and conversations, but the voice processing device 20 may not support either or both of the second processing method and the second communication method. That is, the coded voice data generated by the first processing method in the other voice processing device 20 may be looped back by the first communication method. In this case, the voice data that is looped back to the voice processing device 20 has a lower quality than the voice data that arrives at the other voice processing device 20, but it is possible to check whether the voice data of the user of the voice processing device 20 has arrived at the other voice processing device 20.

[0088] Also, in the above description, an example has been described in which the function for looping back voice data is implemented in the voice processing device 20, but the function may also be implemented in the conference server 30. In this case, the voice processing device 20 can investigate the cause of the trouble by receiving loopback voice data from both the conference server 30 and the other voice processing device 20 and comparing the two voice data. For example, if normal voice data is looped back from the conference server 30 but not from the other voice processing device 20 (or if voice data is looped back but there is a problem with the quality), it is found that there is no problem with the voice processing device 20 or the network line on the voice processing device 20 side, but there is some problem on the other voice processing device 20 side.

[0089] In the above, audio data has been described as an example of data, and encoded audio data has been described as an example of encoded data, but an embodiment of the present invention can also be applied to video data and encoded video data. That is, the audio processing device 20 may have a function for looping back video data. In this case, a video input unit such as a camera is used in addition to or instead of the audio input unit 220 described with reference to FIG. 2, and a video display unit such as a display is used in addition to or instead of the audio output unit 236. With this configuration, it is also possible to check whether the video of the user of the audio processing device 20 reaches another audio processing device 20, and if so, what quality it will reach.

[0090] <6. Hardware configuration> The above describes one embodiment of the present invention. The above-mentioned information processing such as encoding and decoding of audio data is realized by cooperation between software and hardware of the audio processing device 20 described below.

[0091] 7 is a block diagram showing a hardware configuration of the audio processing device 20. The audio processing device 20 includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, and a host bus 204. The audio processing device 20 also includes a bridge 205, an external bus 206, an interface 207, an input device 208, a display device 209, an audio output device 210, a storage device (HDD) 211, a drive 212, and a network interface 215.

[0092] The CPU 201 functions as an arithmetic processing device and a control device, and controls the overall operation of the audio processing device 20 according to various programs. The CPU 201 may be a microprocessor. The ROM 202 stores programs and arithmetic parameters used by the CPU 201. The RAM 203 temporarily stores programs used in the execution of the CPU 201 and parameters that change appropriately in the execution. These are connected to each other by a host bus 204 consisting of a CPU bus and the like. The functions of the first encoding unit 224, the first decoding unit 232, the control unit 240, the forwarding unit 248, the second encoding unit 252, the second communication unit 256, and the like described above can be realized by the cooperation of the CPU 201, the ROM 202, and the RAM 203 with software.

[0093] The host bus 204 is connected to an external bus 206, such as a PCI (Peripheral Component Interconnect / Interface) bus, via a bridge 205. It is not necessary to configure the host bus 204, the bridge 205, and the external bus 206 separately, and these functions may be implemented in a single bus.

[0094] The input device 208 is composed of input means for the user to input information, such as a mouse, keyboard, touch panel, button, microphone, sensor, switch, and lever, and an input control circuit that generates an input signal based on the user's input and outputs it to the CPU 201. The user of the audio processing device 20 can input various data to the audio processing device 20 and instruct the audio processing device 20 to perform processing operations by operating the input device 208.

[0095] The display device 209 includes, for example, a display device such as a liquid crystal display (LCD) device, a projector device, an OLED (Organic Light Emitting Diode) device, a lamp, etc. Furthermore, the audio output device 210 includes an audio output device such as a speaker and a headphone.

[0096] The storage device 211 is a device for storing data configured as an example of a storage unit of the voice processing device 20 according to this embodiment. The storage device 211 may include a storage medium, a recording device for recording data in the storage medium, a reading device for reading data from the storage medium, and a deleting device for deleting data recorded in the storage medium. The storage device 211 is configured, for example, with an HDD (Hard Disk Drive) or an SSD (Solid Storage Drive), or a memory having an equivalent function. This storage device 211 drives the storage and stores the programs executed by the CPU 201 and various data.

[0097] The drive 212 is a reader / writer for a storage medium, and is built into or externally attached to the audio processing device 20. The drive 212 reads out information recorded in a removable storage medium 24, such as an attached magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and outputs the information to the RAM 203 or the storage device 211. The drive 212 can also write information to the removable storage medium 24.

[0098] The network interface 215 is, for example, a communication interface configured with a communication device for connecting to the network 12. In addition, the network interface 215 may be a communication device compatible with a wireless LAN (Local Area Network), or a wired communication device that performs wired communication.

[0099] The hardware configuration of the above-mentioned voice processing device 20 can also be applied to the conference server 30.

[0100] <7. Supplementary Information> Although the preferred embodiment of the present invention has been described in detail above with reference to the accompanying drawings, the present invention is not limited to such an example. It is clear that a person having ordinary knowledge in the technical field to which the present invention pertains can conceive of various modified or altered examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally belong to the technical scope of the present invention.

[0101] For example, each step in the processing of the voice processing device 20 in this specification does not necessarily have to be processed in chronological order according to the order described in the sequence diagram or flowchart. For example, each step in the processing of the voice processing device 20 may be processed in an order different from the order described in the flowchart, or may be processed in parallel.

[0102] It is also possible to create a computer program for causing hardware such as a CPU, a ROM, and a RAM built into the voice processing device 20 to perform functions equivalent to those of the above-described components of the voice processing device 20. Also provided is a storage medium storing the computer program. [Explanation of symbols]

[0103] 20 Audio processing device 220 Audio input section 224 1st encoding section 228 First Communications Department 232 First Decoding Unit 236 Audio output section 238 Operation section 240 Control Unit 244 First Buffer 248 Shuttle Department 252 Second encoding section 256 Second Communications Department 260 Second Decoding Unit 264 Second Buffer 30 Conference Server

Claims

1. a first encoding unit that encodes input data to generate encoded data; a receiving unit that receives the first encoded data from another data processing device; a decoding unit that decodes the first encoded data received by the receiving unit; a second encoding unit that encodes the data obtained by the decoding unit to generate second encoded data; a control unit capable of controlling in parallel the transmission of encoded data generated by the first encoding unit, and controlling the transmission of the second encoded data to the other data processing device when it is determined that the first encoded data is data transmitted from the other data processing device in a specific operation mode; A data processing device comprising:

2. the data processing device further includes an output unit that outputs audio or video based on the data obtained by the decoding unit; 2. The data processing device according to claim 1, wherein the output unit does not output audio and video based on the data obtained by the decoding unit when the control unit determines that the first encoded data is data transmitted from the other data processing device in the specific operating mode.

3. 3. The data processing device according to claim 1, wherein the second encoding unit generates the second encoded data using a second processing method that causes less degradation in quality than a first processing method used to generate the first encoded data.

4. The data processing device according to any one of claims 1 to 3, wherein the control unit controls transmission of the second encoded data using a second communication method that is more reliable than a first communication method used to communicate the first encoded data.

5. The data processing device according to any one of claims 1 to 4, wherein the control unit determines that the first encoded data is data transmitted from the other data processing device in a specific operating mode based on the fact that a predetermined flag is added to the first encoded data.

6. A data processing system having a first data processing device and a second data processing device, The first data processing device comprises: Transmitting first encoded data obtained by encoding data to the second data processing device; The second data processing device comprises: a first encoding unit that encodes input data to generate encoded data; a receiving unit that receives first encoded data from the first data processing device; a decoding unit that decodes the first encoded data received by the receiving unit; a second encoding unit that encodes the data obtained by the decoding unit to generate second encoded data; a control unit capable of simultaneously controlling the transmission of encoded data generated by the first encoding unit and, when it is determined that the first encoded data is data transmitted from the first data processing device in a specific operation mode, controlling the transmission of the second encoded data to the first data processing device; A data processing system comprising:

7. encoding the input data by a first encoding unit to generate encoded data; receiving first encoded data from another data processing device; decoding the first encoded data; and encoding data obtained by decoding the first encoded data by a second encoding unit to generate second encoded data; Controlling the transmission of encoded data generated by the first encoding unit, and controlling the transmission of the second encoded data to the other data processing device when it is determined that the first encoded data is data transmitted from the other data processing device in a specific operation mode, in parallel; 13. A method for processing audio, comprising:

Images