Data processing method, device, electronic device and storage medium

Abstract

The disclosed embodiment discloses a data processing method, device, electronic device and storage medium, the method comprising: acquiring voice data in a collection area; processing the voice data to obtain virtual information, the virtual information comprising text content corresponding to the voice data; outputting the virtual information to at least one display device in a user area, so as to display the virtual information on the display device, so that the user can view the virtual information within the user's field of vision when viewing the collection area through the display device. This technical solution solves the technical problem that some special scenes and / or hearing-impaired people cannot accurately receive voice information when watching live information such as performances by integrating virtual display technology with voice recognition technology.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a data processing method, device, electronic device and storage medium. Background Art

[0002] In some artistic scenes, the needs of hearing-impaired people to watch performances are met by adding display screens to display the actors' lines. However, this solution requires manual operation of each line, which is inefficient, and some viewers may not be able to see the subtitles on the stage clearly. Moreover, this method can only present the content of the lines, and cannot present diversified information on this basis. Summary of the invention

[0003] Embodiments of the present disclosure provide a data processing method, device, electronic device, and computer-readable storage medium.

[0004] In a first aspect, an embodiment of the present disclosure provides a data processing method, including:

[0005] Acquire voice data within the collection area;

[0006] Processing the voice data to obtain virtual information, where the virtual information includes text content corresponding to the voice data;

[0007] The virtual information is output to at least one display device in the user area so as to be displayed on the display device, so that the user can view the virtual information within the user's field of vision when viewing the acquisition area through the display device.

[0008] Further, the voice data is processed to obtain virtual information, including:

[0009] Preprocessing the voice data;

[0010] Using an acoustic model to recognize the preprocessed speech data to obtain corresponding candidate content;

[0011] The candidate content is semantically processed using a semantic model to obtain the text content.

[0012] Furthermore, after the voice data is processed to obtain virtual information, the method further includes:

[0013] The text content is translated into target content corresponding to a target language associated with the display device.

[0014] Furthermore, the display device includes an AR display device.

[0015] In a second aspect, an embodiment of the present disclosure provides a data processing method, which is executed on a display device and includes:

[0016] Acquiring virtual information; the virtual information includes text content obtained by recognizing voice data collected in the collection area;

[0017] The virtual information is displayed so that the user can view the virtual information within the user's field of vision when viewing the acquisition area through the display device.

[0018] Further, displaying the virtual information includes:

[0019] Collecting image data in the acquisition area;

[0020] Acquire a display image by superimposing the virtual information on the image data;

[0021] The display image is displayed.

[0022] Further, the display device includes a transparent display unit, which displays the virtual information, including:

[0023] The text content is displayed on the transparent display unit so that the text content is superimposed on the information viewed by the user through the transparent display unit.

[0024] Furthermore, the method further comprises:

[0025] receiving a target language configured by a user;

[0026] Before displaying the virtual information, the method further includes:

[0027] When the text content in the virtual information does not match the target language, the text content in the virtual information is translated into target content corresponding to the target language.

[0028] In a third aspect, an embodiment of the present disclosure provides a data processing method, which is executed on a display device, wherein the display device includes a voice acquisition unit and a display unit, including:

[0029] Acquiring voice data collected by the voice collection unit;

[0030] Processing the voice data to obtain virtual information, where the virtual information includes text content corresponding to the voice data;

[0031] The virtual information is output to the display unit to display the virtual information on the display unit.

[0032] Further, the voice data is processed to obtain virtual information, including:

[0033] Preprocessing the voice data;

[0034] Using an acoustic model to recognize the preprocessed speech data to obtain corresponding candidate content;

[0035] The candidate content is semantically processed using a semantic model to obtain the text content.

[0036] Further, the display device includes an image acquisition unit, which outputs the virtual information to the display unit so as to display the virtual information on the display unit, including:

[0037] Acquiring image data acquired by the image acquisition unit;

[0038] Acquire a display image by superimposing the virtual information on the image data;

[0039] The display image is output to the display unit for display.

[0040] Further, the display unit includes a transparent display unit, and outputting the virtual information to the display unit to display the virtual information on the display unit includes:

[0041] The text content is output to the transparent display unit for display, so that the text content is superimposed and displayed on the information viewed by the user through the transparent display unit.

[0042] Furthermore, the method further comprises:

[0043] receiving a target language configured by a user;

[0044] Before outputting the virtual information to the display unit, the method further comprises:

[0045] When the text content in the virtual information does not match the target language, the text content in the virtual information is translated into target content corresponding to the target language.

[0046] In a fourth aspect, an embodiment of the present invention provides a data processing device, including:

[0047] A first acquisition module is configured to acquire voice data within a collection area;

[0048] A first processing module is configured to process the voice data to obtain virtual information, where the virtual information includes text content corresponding to the voice data;

[0049] The first output module is configured to output the virtual information to at least one display device in the user area so as to display the virtual information on the display device, so that the user can view the virtual information within the user's field of vision when viewing the acquisition area through the display device.

[0050] In a fifth aspect, an embodiment of the present invention provides a data processing device, the device is located on a display device, and the device includes:

[0051] A second acquisition module is configured to acquire virtual information; the virtual information includes text content obtained by recognizing the voice data collected in the acquisition area;

[0052] The display module is configured to display the virtual information so that the user can view the virtual information within the user's field of vision when viewing the acquisition area through the display device.

[0053] In a sixth aspect, an embodiment of the present invention provides a data processing device, the device is located in a display device, the display device includes a voice collection unit and a display unit, and the device includes:

[0054] A third acquisition module is configured to acquire the voice data collected by the voice collection unit;

[0055] A second processing module is configured to process the voice data to obtain virtual information, where the virtual information includes text content corresponding to the voice data;

[0056] The second output module is configured to output the virtual information to the display unit so as to display the virtual information on the display unit.

[0057] The functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the functions.

[0058] In one possible design, the structure of the above-mentioned device includes a memory and a processor, the memory is used to store one or more computer instructions that support the above-mentioned device to execute the above-mentioned corresponding method, and the processor is configured to execute the computer instructions stored in the memory. The above-mentioned device may also include a communication interface for the above-mentioned device to communicate with other devices or communication networks.

[0059] In the seventh aspect, an embodiment of the present disclosure provides an electronic device, comprising a memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method described in any one of the above aspects.

[0060] In an eighth aspect, an embodiment of the present disclosure provides a computer-readable storage medium for storing computer instructions used by any of the above-mentioned devices, which includes computer instructions involved in executing the method described in any of the above-mentioned aspects.

[0061] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0062] Through the embodiments of the present disclosure, the voice of the target object in the collection area can be converted into text content in real time, and the virtual information including the text content can be sent to the display device for display to the user, so that the user can view the text content corresponding to the voice data in the collection area through the display device while viewing the collection area. The above embodiments solve the technical problem that some special scenes and / or hearing-impaired people cannot accurately receive voice information when watching live information such as performances by integrating virtual display technology with voice recognition technology.

[0063] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Other features, objectives and advantages of the present disclosure will become more apparent through the following detailed description of non-limiting embodiments in conjunction with the accompanying drawings. In the accompanying drawings:

[0065] Figure 1 A flowchart showing a data processing method according to an embodiment of the present disclosure;

[0066] Figure 2 A flowchart showing a data processing method according to another embodiment of the present disclosure;

[0067] Figure 3 A flowchart showing a data processing method according to another embodiment of the present disclosure;

[0068] Figure 4 A schematic diagram of an application process in a stage performance scenario according to an embodiment of the present disclosure is shown;

[0069] Figure 5 It is a schematic diagram of the structure of an electronic device suitable for implementing a data processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0070] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. In addition, for the sake of clarity, parts not related to the description of the exemplary embodiments are omitted in the accompanying drawings.

[0071] In the present disclosure, it should be understood that terms such as "include" or "have" are intended to indicate the presence of features, numbers, steps, behaviors, components, parts, or a combination thereof disclosed in the present specification, and are not intended to exclude the possibility that one or more other features, numbers, steps, behaviors, components, parts, or a combination thereof exist or are added.

[0072] It should also be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0073] The details of the embodiments of the present disclosure are described in detail below through specific examples.

[0074] Figure 1 A flow chart of a data processing method according to an embodiment of the present disclosure is shown. Figure 1 As shown, the data processing method includes the following steps:

[0075] In step S101, voice data in a collection area is acquired;

[0076] In step S102, the voice data is processed to obtain virtual information, where the virtual information includes text content corresponding to the voice data;

[0077] In step S103, the virtual information is output to at least one display device in the user area, so that the virtual information is displayed on the display device, so that the user can view the virtual information within the user's field of vision when viewing the acquisition area through the display device.

[0078] In this embodiment, the data processing method can be implemented on a processor, and the processor can be located on a server or other processing device different from the display device. The acquisition area and the user area can be predetermined, for example, the acquisition area can be the area where the target object that outputs the voice is located, and the user area can be the area where the object that receives the voice is located. For example, in an artistic application scenario, the acquisition area can be the stage area where the performers are located, and the user area can be the area where the audience is located. A voice acquisition device such as a microphone can be set in the acquisition area, and the voice acquisition device outputs the real-time collected voice data to the processor, and the processor converts the voice data into corresponding text content after real-time processing, and sends the virtual information including the text content to the display device in the user area.

[0079] In some embodiments, the virtual information may also include display information of the text content and other related information, and the display information may include display position, display mode, display format, etc. The virtual information may also include synchronization information, such as the occurrence time of the voice data corresponding to the text content.

[0080] In this embodiment, one or more display devices may be included in the user area for use by one or more users. After receiving the virtual information, the display device may display the virtual information within the user's field of vision when the user views the acquisition area through the display device. In some embodiments, the display device may be an AR display device. For example, the display device may be AR glasses. When a user wears AR glasses to view the acquisition area, the virtual information may be displayed on the glasses, so that the user can see the corresponding virtual information while seeing the real picture in the acquisition area through the AR glasses. For example, when the audience watches a stage performance through an AR glasses device, they can see the real performance picture on the stage through the AR glasses, and can also see the corresponding subtitles and other virtual information.

[0081] Through the embodiments of the present disclosure, the voice of the target object in the collection area can be converted into text content in real time, and the virtual information including the text content can be sent to the display device for display to the user, so that the user can view the text content corresponding to the voice data in the collection area through the display device while viewing the collection area. The above embodiments solve the technical problem that some special scenes and / or hearing-impaired people cannot accurately receive voice information when watching live information such as performances by integrating virtual display technology with voice recognition technology.

[0082] In an optional implementation of this embodiment, step S101, i.e., the step of acquiring voice data within the collection area, further includes the following steps:

[0083] The voice data collected in real time by the voice collection device is obtained from the voice collection device arranged in the collection area.

[0084] In this optional implementation, a voice acquisition device may be provided in the acquisition area to acquire voice data in the acquisition area in real time. The voice acquisition device may be, for example, a 360-degree microphone array. After the voice data is acquired, the voice acquisition device may amplify the acquired voice data and perform other pre-processing operations before outputting it to the server. It is understood that the server may be a computer device that is locally configured to process voice data, or a remote server device. The server may be configured according to actual needs and is not limited here.

[0085] In an optional implementation of this embodiment, step S102, i.e., the step of processing the voice data to obtain virtual information, further includes the following steps:

[0086] Preprocessing the voice data;

[0087] Using an acoustic model to recognize the preprocessed speech data to obtain corresponding candidate content;

[0088] The candidate content is semantically processed using a semantic model to obtain the text content.

[0089] In this optional implementation, the voice data can be pre-processed by noise reduction, filtering, etc., and the audio content of the target object in the collection area can be extracted. The target object can be any object that emits speech in the collection area, or one or more objects. The audio content of the target object can be extracted from the pre-processed voice data through functions such as audio recognition, and then the audio content can be subjected to speech recognition using an acoustic model to obtain the corresponding candidate content. The semantic model can then be used to perform contextual semantic processing on the candidate content, and ultimately output text content that conforms to semantic logic. The semantic model can be obtained from lines commonly used in performance scenes and script training. The acoustic model and the semantic model can use models that have been implemented in related technologies, and there is no limitation here.

[0090] In some embodiments, the acquisition area may also be provided with an image acquisition device for acquiring image data synchronized with the voice data. The image data and synchronization information with the voice data may be sent to the server side, and the server side may identify whether the voice data is emitted by the target object in the image data, and perform corresponding preprocessing according to the recognition result. For example, in the application scenario of watching a stage play, if the server side recognizes that the current sound is not emitted by the actor, but the surrounding noise or other personnel based on the image data synchronized with the voice data, then this part of the sound may be filtered out, and only the sound emitted by the actor on the stage may be retained. Of course, it is understandable that, usually, in the application scenario of a stage play, the voice of the actor will be more prominent than the surrounding noise, so it is also possible to filter only the voice data through a commonly used filter, while retaining the voice of the actor. The specific setting may be based on actual needs, and is not limited here.

[0091] In other embodiments, the server can also identify the sound directionality based on the image data synchronized with the voice data, that is, the direction in which the sound emitted by the target object is pointing, and then perform corresponding processing based on the sound directionality. In the case where there are multiple display devices corresponding to the user area, the server can send the processed virtual information to the display device to which the sound is pointing based on the sound directionality, rather than to other display devices. For example, in a performance scene, the actor wants to interact with the audience in the audience and make sounds to the audience in different areas at different times. At this time, the server can send the virtual information to the display devices of the audience in the pointed area based on the current sound directionality of the actor.

[0092] In the application scenario of watching stage performances, the display device can also display the bullet screen information surrounding the actors. Users can add bullet screens through voice control or the setting interface on the display device, or through the user device that interacts with the display device. The display device can also upload the bullet screen added by the user to the server, and the server can share the bullet screen information to other display devices.

[0093] In an optional implementation of this embodiment, after step S102, i.e., the step of processing the voice data to obtain virtual information, the method further includes the following steps:

[0094] The text content is translated into target content corresponding to a target language associated with the display device.

[0095] In this optional implementation, the text content obtained by voice data recognition can be translated into the target content corresponding to the target language. The target language can be a language type associated with the display device, such as Chinese, English, etc. The display device may include multiple, and different display devices may be associated with different target languages. After determining the text content corresponding to the voice data, the text content can be automatically translated into the target content corresponding to the associated target language, and then the virtual information including the target content is output to the corresponding display device. In this way, not only can the problem of special scenes or hearing-impaired people being unable to effectively receive the voice information output by the target object in the acquisition area be solved, but also the problem of language differences between the target object and the user of the display device can be solved. For example, for artistic performance scenes, plays in different languages ​​can be brought to audiences around the world for viewing in this way, greatly reducing the threshold for cultural appreciation.

[0096] Figure 2 A flow chart showing a data processing method according to another embodiment of the present disclosure is shown. Figure 2 As shown, the data processing method includes the following steps:

[0097] In step S201, virtual information is obtained; the virtual information includes text content obtained by recognizing voice data collected in the collection area;

[0098] In step S202, the virtual information is displayed so that the user can view the virtual information within the user's field of vision when viewing the acquisition area through the display device.

[0099] In this embodiment, the data processing method can be implemented on a display device, such as an AR display device. The acquisition area and the user area can be predetermined. For example, the acquisition area can be the area where the target object that outputs the voice is located, and the user area can be the area where the object that receives the voice is located. For example, in an artistic application scenario, the acquisition area can be the stage area where the performers are located, and the user area can be the area where the audience is located. A voice acquisition device such as a microphone can be set in the acquisition area. The voice acquisition device outputs the real-time collected voice data to the corresponding device. The processor on the corresponding device converts the voice data into corresponding text content after real-time processing, and sends the virtual information including the text content to the display device in the user area.

[0100] In some embodiments, the display device may be an AR display device, such as AR glasses.

[0101] The user can view objects in the acquisition area, such as people, objects, scenes, etc., through the AR display device. A display unit can be set on the AR display device. After acquiring virtual information, the virtual information can be displayed on the AR display device, so that when the user views the objects in the acquisition area through the AR display device, the virtual information can be superimposed and displayed within the user's field of vision. For example, when users in the audience area wear AR glasses to watch a stage performance, the voice of the stage performer can be superimposed and displayed on the stage scene viewed by the user through the AR glasses.

[0102] In some embodiments, the virtual information may also include display information of the text content and other related information, such as display location, display mode, display format, etc. The display device may display the text content at an appropriate location based on the display information and other related information in the virtual information, so that the user's viewing of the objects in the acquisition area is not blocked. The display device may also detect the user's viewing angle, binocular distance, etc., and adjust the displayed text size after calculation to avoid user experience problems such as loss of focus or dizziness when viewing subtitles.

[0103] For other details of the embodiments of the present disclosure, please refer to the above Figure 1 The description of the data processing method in the related embodiments will not be repeated here.

[0104] The disclosed embodiments integrate virtual display technology with voice recognition technology to solve the technical problem that some special scenarios and / or hearing-impaired people cannot accurately receive voice information when watching live information such as performances.

[0105] In an optional implementation of this embodiment, step S202, i.e., the step of displaying the virtual information, further includes the following steps:

[0106] Collecting image data in the acquisition area;

[0107] Acquire a display image by superimposing the virtual information on the image data;

[0108] The display image is displayed on the display unit.

[0109] In this optional implementation, the display device also includes an image acquisition unit, such as a monocular camera or a multi-camera. The image data acquired in real time from the acquisition area can be a two-dimensional image or a three-dimensional image. After receiving the virtual information, the display device can perform computer rendering of the text content and the image data in the virtual information to obtain a display image, so as to superimpose the text content on the image data, and display the display image on the display unit. It is understandable that other virtual information can be superimposed on the image data if necessary.

[0110] In an optional implementation of this embodiment, the display device includes a transparent display unit, and step S202, i.e., the step of displaying the virtual information, further includes the following steps:

[0111] The text content is displayed on the transparent display unit so that the text content is superimposed on the information viewed by the user through the transparent display unit.

[0112] In this optional implementation, the display device may be an AR display device, such as AR glasses. The display unit on the display device may be a transparent display unit. When the user wears the display device, the transparent display unit may be located at the eye position, so that the user can view the information in the environment through the transparent unit. The transparent display unit also has a display function, which can display virtual information, such as text content, so that when the user views the information in the collection area through the transparent display unit, the text content can be superimposed on the viewed information. Since the natural reflected light in the surrounding environment can pass through the transparent display unit normally, the user can view the surrounding environment and objects through the transparent display unit without affecting the wearer's line of sight. In this way, the information in the collection area viewed by the user is real, and at the same time, the text content corresponding to the voice data emitted by the object in the collection area can be seen without any intervention. This method is particularly suitable for artistic scenes, and can realize the automatic collection and display of subtitles without destroying the artistic performance.

[0113] In an optional implementation of this embodiment, the method further includes the following steps:

[0114] receiving a target language configured by a user;

[0115] Before displaying the virtual information, the method further includes:

[0116] When the text content in the virtual information does not match the target language, the text content in the virtual information is translated into target content corresponding to the target language.

[0117] In this optional implementation, the user can configure the target language for the display device through the client. When the text content in the received virtual information does not match the target language configured by the user, the text content in the virtual information can be translated into the target content corresponding to the target language and then displayed on the display device. In this way, the display device can be suitable for target users who use any language.

[0118] In some embodiments, the user can set the language category on the display device or a user device that interacts with the display device, such as a mobile phone. That is, multiple optional language categories can be pre-set on the display device, and the user can select the corresponding category according to the language he is familiar with. When the text content in the received virtual information is inconsistent with the language category selected by the user, the display device automatically translates the text content into the language category selected by the user.

[0119] In other embodiments, the user may also wear headphones to play corresponding voice data on the headphones while displaying virtual information on the display device. The user may also set a language category on the headphones, and when the received voice data is inconsistent with the selected language category, the voice data may be automatically translated into the language category selected by the user. Of course, it is understandable that the automatic translation process may be completed on the server side. After the user selects the configured language category through the headphones, the headphones send the configured language category to the server, and the server may translate the voice data into the voice corresponding to the configured language category and send it to the headphones during use.

[0120] Figure 3 A flow chart showing a data processing method according to another embodiment of the present disclosure is shown. Figure 2 As shown, the data processing method includes the following steps:

[0121] In step S301, the voice data collected by the voice collection unit is obtained;

[0122] In step S302, the voice data is processed to obtain virtual information, where the virtual information includes text content corresponding to the voice data;

[0123] In step S303, the virtual information is output to the display unit so as to display the virtual information on the display unit.

[0124] In this embodiment, the data processing method can be implemented on a display device, such as an AR display device. The display device may include a voice acquisition unit and a display unit. The voice acquisition unit may be, for example, a microphone array. The display device may be glasses, and the display unit may be arranged on the lenses. During the use of the display device, the voice acquisition unit may collect voice data in the surrounding environment in real time, and output the collected voice data to the processing unit on the display device. The processing unit obtains virtual information by processing the voice data. The virtual information may include text content corresponding to the voice data. The virtual information is output to the display unit and then displayed. The acquisition area and the user area may be predetermined. For example, the acquisition area may be the area where the target object for outputting the voice is located, and the user area may be the area where the object for receiving the voice is located.

[0125] The user can view information in the environment, such as people, objects, scenes, etc., through the display unit on the display device. When the user views the information in the environment through the display unit, the virtual information can be superimposed and displayed within the user's field of vision. For example, when the user wears glasses to watch a stage performance, the voice of the stage performer can be superimposed and displayed on the stage scene viewed by the user through the glasses.

[0126] In some embodiments, the virtual information may also include display information of the text content and other related information, such as display location, display mode, display format, etc. The display device may display the text content at an appropriate location based on the display information and other related information in the virtual information, so that the user's viewing of the objects in the acquisition area is not blocked. The display device may also detect the user's viewing angle, binocular distance, etc., and adjust the displayed text size after calculation to avoid user experience problems such as loss of focus or dizziness when viewing subtitles.

[0127] Through the embodiments of the present disclosure, the collected voice can be converted into text content in real time by using a display device, and virtual information including text content can be displayed to the user, so that the user can view the text content corresponding to the voice data emitted by the object in the environment through the display device while viewing the information in the environment. The above embodiments solve the technical problem that some special scenes and / or hearing-impaired people cannot accurately receive voice information when watching live information such as performances by integrating virtual display technology with voice recognition technology.

[0128] In an optional implementation of this embodiment, step S302, i.e., the step of processing the voice data to obtain virtual information, further includes the following steps:

[0129] Preprocessing the voice data;

[0130] Using an acoustic model to recognize the preprocessed speech data to obtain corresponding candidate content;

[0131] The candidate content is semantically processed using a semantic model to obtain the text content.

[0132] In this optional implementation, the voice data can be pre-processed by noise reduction, filtering, etc., and the audio content of the target object in the collection area can be extracted. The target object can be any object that emits speech in the collection area, or one or more objects. The audio content of the target object can be extracted from the pre-processed voice data through functions such as audio recognition, and then the audio content can be subjected to speech recognition using an acoustic model to obtain the corresponding candidate content. The semantic model can then be used to perform contextual semantic processing on the candidate content, and ultimately output text content that conforms to semantic logic. The semantic model can be obtained from lines commonly used in performance scenes and script training. The acoustic model and the semantic model can use models that have been implemented in related technologies, and there is no limitation here.

[0133] In an optional implementation of this embodiment, the display device includes an image acquisition unit, and step S302, that is, the step of outputting the virtual information to the display unit to display the virtual information on the display unit, further includes the following steps:

[0134] Acquiring image data acquired in real time by the image acquisition unit;

[0135] Acquire a display image by superimposing the virtual information on the image data;

[0136] The display image is output to the display unit for display.

[0137] In this optional implementation, the display device also includes an image acquisition unit, such as a monocular camera or a multi-camera. The image acquisition unit is used to acquire image data in the environment in real time, and the image data can be a two-dimensional image or a three-dimensional image. After the processing unit obtains the corresponding virtual information using the voice data collected by the voice acquisition unit, the text content in the virtual information and the image data can be computer rendered to obtain a display image, so that the text content is superimposed on the image data, and the display image is displayed on the display unit. It is understandable that other virtual information can also be superimposed on the image data if necessary.

[0138] In an optional implementation of this embodiment, the display unit includes a transparent display unit, and step S302, that is, the step of outputting the virtual information to the display unit to display the virtual information on the display unit, further includes the following steps:

[0139] The text content is displayed on the transparent display unit so that the text content is superimposed on the information viewed by the user through the transparent display unit.

[0140] In this optional implementation, the display unit on the display device can be a transparent display unit. When the user wears the display device, the transparent display unit can be located in the eye area, so that the user can view the information in the environment through the transparent unit. The transparent display unit also has a display function and can display virtual information, such as text content, so that when the user views the information in the collection area through the transparent display unit, the text content can be superimposed on the viewed information. Since the natural reflected light in the surrounding environment can pass through the transparent display unit normally, the user can view the surrounding environment and objects through the transparent display unit without affecting the wearer's line of sight. In this way, the information in the collection area viewed by the user is real, and at the same time, the text content corresponding to the voice data emitted by the object in the environment can be seen without any intervention. This method is particularly suitable for artistic scenes and can realize the automatic collection and display of subtitles without destroying the artistic performance.

[0141] In an optional implementation of this embodiment, the method further includes the following steps:

[0142] receiving a target language configured by a user;

[0143] Before outputting the virtual information to the display unit, the method further comprises:

[0144] When the text content in the virtual information does not match the target language, the text content in the virtual information is translated into target content corresponding to the target language.

[0145] In this optional implementation, the user can configure the target language for the display device through the client. When the text content in the received virtual information does not match the target language configured by the user, the text content in the virtual information can be translated into the target content corresponding to the target language and then displayed on the display device. In this way, the display device can be suitable for target users who use any language.

[0146] Figure 4 FIG. 1 is a schematic diagram showing an application process in a stage performance scenario according to an embodiment of the present disclosure. Figure 4As shown, a 360-degree microphone array 401 is arranged around the stage for collecting voice data emitted by performers on the stage in real time. The processing device 402 can be arranged in the space where the stage is located, or can be arranged remotely to communicate with the microphone array 401 through a communication network. In the audience area, the audience can wear AR glasses 403 to watch the performance on the stage. The AR glasses can communicate with the processing device 402 through a communication network. During the performance, the microphone array 401 sends the real-time collected voice data to the processing device 402 through the network. After being processed by the processing device 402, subtitle information can be obtained. The subtitle information is sent to the AR glasses 403 worn by the audience in the audience area through the network, and is displayed on the AR glasses 403 in real time, so that the audience can overlay subtitles on the AR glasses 403 while watching the object performance.

[0147] The following are embodiments of the apparatus of the present disclosure, which can be used to execute embodiments of the method of the present disclosure.

[0148] According to a data processing device in one embodiment of the present disclosure, the device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. The data processing device includes:

[0149] A first acquisition module is configured to acquire voice data within a collection area;

[0150] A first processing module is configured to process the voice data to obtain virtual information, where the virtual information includes text content corresponding to the voice data;

[0151] The first output module is configured to output the virtual information to at least one display device in the user area so as to display the virtual information on the display device, so that the user can view the virtual information within the user's field of vision when viewing the acquisition area through the display device.

[0152] In an optional implementation of this embodiment, the first acquisition module includes:

[0153] The first acquisition submodule is configured to acquire the voice data collected by the voice acquisition device from the voice acquisition device arranged in the acquisition area.

[0154] In an optional implementation of this embodiment, the first processing module includes:

[0155] A first preprocessing submodule is configured to preprocess the voice data;

[0156] A first recognition submodule is configured to use an acoustic model to recognize the pre-processed voice data to obtain corresponding candidate content;

[0157] The first semantic processing submodule is configured to perform semantic processing on the candidate content using a semantic model to obtain the text content.

[0158] In an optional implementation of this embodiment, after the first processing module, the device further includes:

[0159] The first translation module is configured to translate the text content into target content corresponding to a target language associated with the display device.

[0160] In an optional implementation of this embodiment, the display device includes an AR display device.

[0161] The data processing device in this embodiment Figure 1 The data processing methods in the embodiment shown and the related embodiments correspond to each other. For details, please refer to the above Figure 1 The description of the data processing method in the illustrated embodiment and related embodiments will not be repeated here.

[0162] According to another embodiment of the present disclosure, a data processing device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. The device is located on a display device, and the data processing device includes:

[0163] A second acquisition module is configured to acquire virtual information; the virtual information includes text content obtained by recognizing the voice data collected in the acquisition area;

[0164] The display module is configured to display the virtual information so that the user can view the virtual information within the user's field of vision when viewing the acquisition area through the display device.

[0165] In an optional implementation of this embodiment, the display module includes:

[0166] A collection submodule, configured to collect image data within the collection area;

[0167] A second acquisition submodule is configured to acquire a display image by superimposing the virtual information on the image data;

[0168] The first display submodule is configured to display the display image.

[0169] In an optional implementation of this embodiment, the display device includes a transparent display unit, and the display module includes:

[0170] The second display submodule is configured to display the text content on the transparent display unit so that the text content is superimposed on the information viewed by the user through the transparent display unit.

[0171] In an optional implementation of this embodiment, the method further includes:

[0172] A first receiving module is configured to receive a target language configured by a user;

[0173] Before the display module, the method further comprises:

[0174] The second translation module is configured to translate the text content in the virtual information into target content corresponding to the target language when the text content in the virtual information does not match the target language.

[0175] The data processing device in this embodiment Figure 2 The data processing methods in the embodiment shown and the related embodiments correspond to each other. For details, please refer to the above Figure 2 The description of the data processing method in the illustrated embodiment and related embodiments will not be repeated here.

[0176] According to another embodiment of the present disclosure, a data processing device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. The device is located in a display device, which includes a voice acquisition unit and a display unit. The data processing device includes:

[0177] A third acquisition module is configured to acquire the voice data collected by the voice collection unit;

[0178] A second processing module is configured to process the voice data to obtain virtual information, where the virtual information includes text content corresponding to the voice data;

[0179] The second output module is configured to output the virtual information to the display unit so as to display the virtual information on the display unit.

[0180] In an optional implementation of this embodiment, the second processing module includes:

[0181] A second preprocessing submodule is configured to preprocess the voice data;

[0182] A second recognition submodule is configured to use an acoustic model to recognize the pre-processed voice data to obtain corresponding candidate content;

[0183] The second semantic processing submodule is configured to perform semantic processing on the candidate content using a semantic model to obtain the text content.

[0184] In an optional implementation of this embodiment, the display device includes an image acquisition unit, and the second output module includes:

[0185] A third acquisition submodule is configured to acquire the image data acquired by the image acquisition unit;

[0186] A fourth acquisition submodule is configured to acquire a display image by superimposing the virtual information on the image data;

[0187] The first output submodule is configured to output the display image to the display unit for display.

[0188] In an optional implementation of this embodiment, the display unit includes a transparent display unit, and the second output module includes:

[0189] The second output submodule is configured to output the text content to the transparent display unit for display, so that the text content is superimposed on the information viewed by the user through the transparent display unit.

[0190] In an optional implementation of this embodiment, the device further includes:

[0191] A second receiving module is configured to receive a target language configured by a user;

[0192] Before the second output module, the device further includes:

[0193] The third translation module is configured to translate the text content in the virtual information into target content corresponding to the target language when the text content in the virtual information does not match the target language.

[0194] The data processing device in this embodiment Figure 3 The data processing methods in the embodiment shown and the related embodiments correspond to each other. For details, please refer to the above Figure 3 The description of the data processing method in the illustrated embodiment and related embodiments will not be repeated here.

[0195] Figure 5 It is a schematic diagram of the structure of an electronic device suitable for implementing the data processing method according to an embodiment of the present disclosure.

[0196] like Figure 5As shown, the electronic device 500 includes a processing unit 501, which can be implemented as a processing unit such as a CPU, a GPU, an FPGA, and an NPU. The processing unit 501 can perform various processes in the embodiments of any of the above methods of the present disclosure according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 into a random access memory (RAM) 503. In RAM503, various programs and data required for the operation of the electronic device 500 are also stored. The processing unit 501, ROM502, and RAM503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0197] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage section 508 as needed.

[0198] In particular, according to an embodiment of the present disclosure, any method in the above referenced embodiment of the present disclosure may be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program tangibly contained on a readable medium thereof, and the computer program includes a program code for executing any method in the embodiment of the present disclosure. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 509, and / or installed from a removable medium 511.

[0199] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the road map or block diagram can represent a part of a module, program segment or code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a different order from the order marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0200] The units or modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The units or modules described may also be set in a processor, and the names of these units or modules do not constitute limitations on the units or modules themselves in some cases.

[0201] As another aspect, the present disclosure further provides a computer-readable storage medium, which may be a computer-readable storage medium included in the device described in the above embodiment; or a computer-readable storage medium that exists independently and is not assembled into the device. The computer-readable storage medium stores one or more programs, and the programs are used by one or more processors to execute the method described in the present disclosure.

[0202] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present disclosure is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other.

Claims

1. A data processing method, wherein: include: Acquire voice data in a collection area and image data synchronized with the voice data; Processing the voice data to obtain virtual information, the virtual information including text content corresponding to the voice data; and identifying sound directivity based on image data synchronized with the voice data; The virtual information is output to at least one display device to which the sound in the user area is directed, so that the virtual information is displayed on the display device, so that the user can view the virtual information within the user's field of vision while viewing the real picture in the acquisition area through the display device.

2. The method according to claim 1, wherein: Processing the voice data to obtain virtual information includes: Preprocessing the voice data; Using an acoustic model to recognize the preprocessed speech data to obtain corresponding candidate content; The candidate content is semantically processed using a semantic model to obtain the text content.

3. The method according to claim 1 or 2, wherein: After the voice data is processed to obtain virtual information, the method further includes: The text content is translated into target content corresponding to a target language associated with the display device.

4. The method according to claim 1 or 2, wherein: The display device includes an AR display device.

5. A data processing method, wherein: The method is performed on a display device and includes: Acquiring virtual information; the virtual information includes text content obtained by recognizing voice data collected in the collection area; Displaying the virtual information so that the user can view the virtual information within the user's field of vision while viewing the real image in the acquisition area through the display device; The display device is a display device determined by the server according to the image data identifying the sound directionality, and the image data is data collected synchronously with the voice data.

6. The method according to claim 5, wherein: Displaying the virtual information includes: Collecting image data in the acquisition area; Acquire a display image by superimposing the virtual information on the image data; The display image is displayed.

7. The method according to claim 5 or 6, wherein: The display device includes a transparent display unit, which displays the virtual information, including: The text content is displayed on the transparent display unit so that the text content is superimposed on the information viewed by the user through the transparent display unit.

8. The method according to claim 5 or 6, wherein: The method further comprises: receiving a target language configured by a user; Before displaying the virtual information, the method further includes: When the text content in the virtual information does not match the target language, the text content in the virtual information is translated into target content corresponding to the target language.

9. A data processing method, wherein: The method is performed on a display device, wherein the display device includes a voice collection unit and a display unit, including: Acquiring voice data collected by the voice collection unit; Processing the voice data to obtain virtual information, where the virtual information includes text content corresponding to the voice data; outputting the virtual information to the display unit to display the virtual information on the display unit; Wherein, the display unit includes a transparent display unit, and outputting the virtual information to the display unit so as to display the virtual information on the display unit includes: Outputting the text content to the transparent display unit for display, so that the text content is superimposed on the information viewed by the user through the transparent display unit; The display device is a display device determined by the server according to the image data identifying the sound directivity, and the image data is data collected synchronously with the voice data.

10. The method according to claim 9, wherein: Processing the voice data to obtain virtual information includes: Preprocessing the voice data; Using an acoustic model to recognize the preprocessed speech data to obtain corresponding candidate content; The candidate content is semantically processed using a semantic model to obtain the text content.

11. The method according to claim 9 or 10, wherein: The display device includes an image acquisition unit, which outputs the virtual information to the display unit so as to display the virtual information on the display unit, including: Acquiring image data acquired by the image acquisition unit; Acquire a display image by superimposing the virtual information on the image data; The display image is output to the display unit for display.

12. The method according to claim 9 or 10, wherein: The method further comprises: receiving a target language configured by a user; Before outputting the virtual information to the display unit, the method further comprises: When the text content in the virtual information does not match the target language, the text content in the virtual information is translated into target content corresponding to the target language.

13. A data processing device, wherein: include: A first acquisition module is configured to acquire voice data in a collection area and image data synchronized with the voice data; A first processing module is configured to process the voice data to obtain virtual information, wherein the virtual information includes text content corresponding to the voice data; and recognize sound directivity according to image data synchronized with the voice data; The first output module is configured to output the virtual information to at least one display device to which the sound in the user area is directed, so as to display the virtual information on the display device, so that the user can view the virtual information within the user's field of vision while viewing the real picture in the acquisition area through the display device.

14. A data processing device, wherein: The device is located on a display device, and comprises: A second acquisition module is configured to acquire virtual information; the virtual information includes text content obtained by recognizing the voice data collected in the acquisition area; A display module is configured to display the virtual information so that the user can view the virtual information within the user's field of vision while viewing the real image in the acquisition area through the display device; The display device is a display device determined by the server according to the image data identifying the sound directionality, and the image data is data collected synchronously with the voice data.

15. A data processing device, wherein: The device is located in a display device, the display device includes a voice collection unit and a display unit, and the device includes: A third acquisition module is configured to acquire the voice data collected by the voice collection unit; A second processing module is configured to process the voice data to obtain virtual information, where the virtual information includes text content corresponding to the voice data; a second output module, configured to output the virtual information to the display unit so as to display the virtual information on the display unit; Wherein, the display unit includes a transparent display unit, and the second output module includes: A second output submodule is configured to output the text content to the transparent display unit for display, so that the text content is superimposed on the information viewed by the user through the transparent display unit; The display device is a display device determined by the server according to the image data identifying the sound directivity, and the image data is data collected synchronously with the voice data.

16. An electronic device, wherein: comprising a memory and a processor; wherein, The memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method according to any one of claims 1 to 12.

17. A computer-readable storage medium having computer instructions stored thereon, wherein: When the computer instructions are executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images