Audio and video scene reconstruction method and device, equipment and storage medium

Abstract

The invention discloses an audio and video scene reconstruction method and device, equipment and a storage medium, and relates to the technical field of computer vision, and the method comprises the steps: obtaining a scene depth image of a target scene through a camera array; establishing a scene model based on the scene depth image; recognizing a face image in the scene depth image, and extracting face features in the face image; fitting the face features with a preset face model to obtain a face model corresponding to the face image; and mapping the face model to the scene model based on the position information of the face image in the scene depth image to obtain a complete scene model. According to the invention, the scene information of the conference and the language and face information of each participant can be accurately stored, so that the conference can be recorded more completely and accurately.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method, device, equipment and storage medium for reconstructing an audio and video scene. Background Art

[0002] In the modern office environment, meetings have become an important form of team collaboration and knowledge sharing. In order to ensure the integrity and traceability of meeting content, it is particularly important to accurately record meeting scenes.

[0003] At present, the recording of meeting scenes mainly relies on video and audio equipment. Video recording can capture the visual information of the meeting, including the expressions, movements and meeting environment of the participants; while audio recording can record the voice content of the meeting, including the key points and details of the discussion. These recording methods meet the basic needs of meeting records to a certain extent.

[0004] However, in practical applications, both video and audio recording methods have the problem of difficulty in completely restoring the conference scene. Although video recording can capture rich visual information, it is prone to blurry, blocked or missed images in situations such as insufficient light, limited angles or a large number of participants. Although audio recording can clearly convey the voice content, it lacks intuitive visual information and is difficult to accurately reflect the on-site atmosphere of the meeting and the emotional changes of the participants. In addition, video and audio recordings are easily affected by factors such as equipment performance, environmental factors and human operational errors, resulting in uneven recording quality. Moreover, in meetings where it is necessary to record the movement changes of people or scenes, it is difficult to meet the needs of high-quality meeting records. Therefore, how to achieve complete and accurate recording of conference scenes has become a technical problem that needs to be solved urgently.

[0005] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0006] The main purpose of this application is to provide an audio and video scene reconstruction method, device, equipment and storage medium, aiming to solve the technical problem of how to achieve complete and accurate recording of conference scenes.

[0007] To achieve the above objectives, the present application proposes a method for reconstructing an audio and video scene, the method comprising: Acquire a scene depth image of the target scene through a camera array; Establishing a scene model based on the scene depth image; Identifying a face image in the scene depth image, and extracting face features in the face image; Fitting the facial features to a preset facial model to obtain a facial model corresponding to the facial image; Based on the position information of the face image in the scene depth image, the face model is mapped to the scene model to obtain a complete scene model.

[0008] In one embodiment, the step of establishing a scene model based on the scene depth image comprises: Traversing each of the scene depth images, and aligning the scene depth images to a preset reference coordinate system; Convert each pixel in the scene depth image into a point in the reference coordinate system to obtain a three-dimensional point cloud composed of each scene depth image; Meshing the three-dimensional point cloud to obtain a mesh model; The scene depth image is mapped to the grid model to obtain scene modeling.

[0009] In one embodiment, after the step of fitting the facial features with a preset facial model to obtain a facial model corresponding to the facial image, the method further includes: Tracking each of the facial features in each frame image after the scene depth image; The face model is updated in real time according to each of the tracked face features.

[0010] In one embodiment, the method further comprises: Identifying a torso image corresponding to the face image in the scene depth image; Tracking torso features of the torso image in real time; The torso features are mapped to a preset torso basic model to obtain a torso model corresponding to the face image.

[0011] In one embodiment, the method further comprises: Acquire sound signals through a microphone array; A voice signal in the sound signal is recognized, and the voice signal is converted into text information.

[0012] In one embodiment, after the step of converting the voice signal into text information, the method further includes: Identifying the sound source location of the sound signal in the target scene by using the microphone array; Determining a target face image in each of the face images according to the sound source localization; Determine a display position of a text box according to the face position of the target face image in the complete scene model, wherein the distance between the display position and the face position is less than a preset threshold; The text information is displayed in the text box.

[0013] In addition, to achieve the above-mentioned purpose, the present application also proposes an audio and video scene reconstruction device, the audio and video scene reconstruction device comprising: An image acquisition module, used to acquire a scene depth image of a target scene through a camera array; A scene modeling module, used to establish a scene model based on the scene depth image; A face recognition module, used to recognize a face image in the scene depth image and extract face features in the face image; A face modeling module, used for fitting the face features with a preset face model to obtain a face model corresponding to the face image; A scene fusion module is used to map the face model to the scene model based on the position information of the face image in the scene depth image to obtain a complete scene model.

[0014] In addition, to achieve the above-mentioned purpose, the present application also proposes an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the audio and video scene reconstruction method as described above.

[0015] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the audio and video scene reconstruction method described above are implemented.

[0016] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the audio and video scene reconstruction method described above are implemented.

[0017] The present application provides a method for reconstructing an audio and video scene. In the conference, the present application obtains a scene depth image of a conference target scene through a camera array, then models the current conference scene based on the multiple scene depth images obtained to obtain a scene model, then identifies face images in each scene depth image, extracts face features, and then fits the face features with a pre-set face model to obtain a corresponding face model. The face model is mapped to the scene model to obtain a complete conference scene model.

[0018] In summary, the present application can capture multi-angle scene depth images of the target scene through a camera array, and then establish a scene model based on three-dimensional reconstruction technology to restore the scene information of the meeting. Moreover, in the scene depth image, the face recognition technology is used to accurately identify the face image, and the extracted face features are fitted with the preset face model to generate a face model that is highly similar to the real face. Based on the position information of the face image in the scene depth image, the generated face model can be accurately mapped to the scene model, thereby constructing a complete scene model that includes both the meeting scene background and the real participants' faces. Compared with meeting records that only include video or audio, by modeling the meeting scene and the participants' faces, the scene information of the meeting and the language and facial information of each participant can be accurately saved, so that the meeting can be recorded more completely and accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 A flowchart of the first embodiment of the audio and video scene reconstruction method of the present application is provided; Figure 2 A flowchart of the second embodiment of the audio and video scene reconstruction method of the present application is provided; Figure 3 A flowchart of the third embodiment of the audio and video scene reconstruction method of the present application is provided; Figure 4 This is a schematic diagram of the module structure of the audio and video scene reconstruction device according to an embodiment of the present application; Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the audio and video scene reconstruction method in the embodiment of the present application.

[0022] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0023] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0024] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0025] The main solution of the embodiment of the present application is: acquiring a scene depth image of the target scene through a camera array; establishing a scene model based on the scene depth image; identifying a face image in the scene depth image, and extracting face features in the face image; fitting the face features with a preset face model to obtain a face model corresponding to the face image; and mapping the face model to the scene model based on the position information of the face image in the scene depth image to obtain a complete scene model.

[0026] In this embodiment, for the convenience of description, the electronic device is used as the execution subject for explanation below.

[0027] In the modern office environment, meetings have become an important form of team collaboration and knowledge sharing. In order to ensure the integrity and traceability of meeting content, it is particularly important to accurately record meeting scenes.

[0028] At present, the recording of meeting scenes mainly relies on video and audio equipment. Video recording can capture the visual information of the meeting, including the expressions, movements and meeting environment of the participants; while audio recording can record the voice content of the meeting, including the key points and details of the discussion. These recording methods meet the basic needs of meeting records to a certain extent.

[0029] However, in practical applications, both video and audio recording methods have the problem of difficulty in completely restoring the conference scene. Although video recording can capture rich visual information, it is prone to blurry, blocked or missed images in situations such as insufficient light, limited angles or a large number of participants. Although audio recording can clearly convey the voice content, it lacks intuitive visual information and is difficult to accurately reflect the on-site atmosphere of the meeting and the emotional changes of the participants. In addition, video and audio recordings are easily affected by factors such as equipment performance, environmental factors and human operational errors, resulting in uneven recording quality. Moreover, in meetings where it is necessary to record the movement changes of people or scenes, it is difficult to meet the needs of high-quality meeting records. Therefore, how to achieve complete and accurate recording of conference scenes has become a technical problem that needs to be solved urgently.

[0030] In response to the above problems, the present application provides an audio and video scene reconstruction method. In the meeting, the present application obtains a scene depth image of the target scene of the meeting through a camera array, and then models the current meeting scene based on the multiple scene depth images obtained to obtain a scene model. The face images in each scene depth image are then identified, and the features of the face are extracted. The face features are then fit with a pre-set face model to obtain a corresponding face model. The face model is mapped to the scene model to obtain a complete meeting scene model.

[0031] In summary, the present application can capture multi-angle scene depth images of the target scene through a camera array, and then establish a scene model based on three-dimensional reconstruction technology to restore the scene information of the meeting. Moreover, in the scene depth image, the face recognition technology is used to accurately identify the face image, and the extracted face features are fitted with the preset face model to generate a face model that is highly similar to the real face. Based on the position information of the face image in the scene depth image, the generated face model can be accurately mapped to the scene model, thereby constructing a complete scene model that includes both the meeting scene background and the real participants' faces. Compared with meeting records that only include video or audio, by modeling the meeting scene and the participants' faces, the scene information of the meeting and the language and facial information of each participant can be accurately saved, so that the meeting can be recorded more completely and accurately.

[0032] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, etc. The following takes an electronic device as an example to illustrate this embodiment and the following embodiments.

[0033] Based on this, the embodiment of the present application provides a method for reconstructing an audio and video scene, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the audio and video scene reconstruction method of the present application.

[0034] In this embodiment, the audio and video scene reconstruction method includes steps S10 to S50: Step S10, acquiring a scene depth image of the target scene through a camera array; It should be noted that in this embodiment, the camera array can be an array of ordinary cameras, which acquires the depth information of the scene by collecting multiple images, or a TOF (Time of Flight) camera array, which directly acquires the scene depth information from multiple angles. The scene depth image includes the depth information of the image and each pixel in the image.

[0035] In this embodiment, the configured camera array is used to capture light information of the target scene from different angles and distances. Through depth perception technology, such as structured light, stereo vision or ToF (Time of Flight), the camera array can calculate the distance from each pixel to the camera, thereby generating a scene depth image containing depth information.

[0036] Step S20, establishing a scene model based on the scene depth image; In this embodiment, using computer vision and 3D reconstruction technology, the device constructs a 3D model of the target scene based on the distance information and light and shadow information in the depth image. This model not only includes the shape and size of the objects in the scene, but also reflects the spatial relationship between them.

[0037] Further, in a feasible implementation manner, the above-mentioned step S20 includes steps S21 to S24: Step S21, traversing each of the scene depth images, and aligning the scene depth images to a preset reference coordinate system; In this embodiment, when modeling a scene based on a scene depth image, firstly, all scene depth images captured by the camera array are traversed, and for each depth image, image registration or camera calibration technology is used to align it to a preset unified reference coordinate system.

[0038] Step S22, converting each pixel point in the scene depth image into a point in the reference coordinate system to obtain a three-dimensional point cloud composed of each scene depth image; In this embodiment, after the images are aligned, each pixel in each depth image is converted to its three-dimensional coordinate point in the reference coordinate system. This conversion process takes into account the depth value (i.e., distance information) of the pixel point, thereby obtaining a three-dimensional point cloud composed of all depth images. Each point in the point cloud represents a specific spatial position in the scene.

[0039] Step S23, meshing the three-dimensional point cloud to obtain a mesh model; In this embodiment, after obtaining the three-dimensional point cloud, a three-dimensional reconstruction algorithm, such as Delaunay triangulation, Poisson surface reconstruction or Marching Cubes algorithm, is used to mesh the obtained three-dimensional point cloud, and the discrete point cloud data is converted into a continuous mesh model. The surface of the mesh model is composed of a series of triangular or quadrilateral meshes, which can accurately reflect the shape and details of the scene.

[0040] Step S24, mapping the scene depth image to the grid model to obtain scene modeling.

[0041] In this embodiment, after the mesh model is constructed, the original scene depth image is mapped onto the surface of the mesh model, retaining the texture and color information in the depth image, so that the mesh model not only has an accurate three-dimensional shape but also has a realistic visual effect.

[0042] The above steps convert the scene depth image into a high-quality scene model through a series of 3D reconstruction and mapping technologies, thus restoring the scene environment of the meeting.

[0043] Step S30, identifying a face image in the scene depth image, and extracting face features in the face image; In this embodiment, after the scene is modeled, a face recognition algorithm is used to locate and identify the face area in the depth image. Subsequently, key facial features, such as the shape and position of the eyes, nose, and mouth, are extracted from the face image through feature extraction techniques, such as deep learning networks or feature point matching.

[0044] Step S40, fitting the facial features with a preset facial model to obtain a facial model corresponding to the facial image; In this embodiment, the extracted facial features are compared and fitted with the models in the pre-stored facial model library. Through an optimization algorithm, such as the least squares method or the iterative closest point (ICP) algorithm, the device finds the facial model that best matches the extracted features and generates a corresponding facial model.

[0045] Specifically, a series of facial model libraries are pre-stored in the device. These facial models are not set randomly, but are scientifically preset based on a variety of typical facial features obtained from big data analysis. Specifically, these preset facial models fully refer to the widely collected and statistically analyzed facial data, covering a variety of facial variations from common oval faces to square faces, heart-shaped faces, etc., ensuring the broad representativeness and practicality of the model library.

[0046] When receiving a face image to be processed, the image processing algorithm is first used to accurately extract the facial feature points, including but not limited to the corners of the eyes, corners of the mouth, nose tip and other key facial landmarks. Then, based on the relative position relationship between these feature points, intelligent matching is performed among multiple pre-stored facial models to find the facial model that is most similar to the current facial features (i.e., the highest similarity) as the initial reference model.

[0047] After the initial model is selected, the precise positional relationship between the feature points is further used to carefully adjust and optimize the selected facial model. This process involves geometric transformation and morphological processing, aiming to ensure that the contour, proportion and detailed features of the facial model can be perfectly aligned and fitted with the feature points of the target face, thereby achieving a highly personalized customization effect.

[0048] Finally, after the facial model is finely adjusted, the original face image is accurately mapped to the fitted personalized facial model through texture mapping technology. This process not only retains all the visual details of the original face image, but also enhances the three-dimensional sense and realism of the face image through the three-dimensional structure and texture information of the model, thus successfully generating a high-precision face model corresponding to the input face image.

[0049] Furthermore, in a feasible implementation manner, after the above step S40, the method may further include steps A10 to A20: Step A10, tracking each of the facial features in each frame image after the scene depth image; In this embodiment, after successfully fitting the face model, the subsequent image frames in the target scene are continuously monitored and captured. The face features in these subsequent image frames are tracked in real time using a face tracking algorithm, such as Kalman filtering, optical flow method or deep learning tracker. These features are the same as the features previously extracted and used to fit the face model, including the shape and position of key parts such as eyes, nose, mouth, etc.

[0050] Step A20: updating the face model in real time according to the tracked face features.

[0051] In this embodiment, based on the tracked facial features, the previously fitted facial model is updated in real time, and this updating process includes adjusting the shape, texture or expression of the facial model to better match the facial features in the current image frame. The updating algorithm may include techniques such as shape interpolation, texture mapping or expression transfer to ensure that the facial model changes synchronously with the real face.

[0052] Step S50: Mapping the face model to the scene model based on the position information of the face image in the scene depth image to obtain a complete scene model.

[0053] In this embodiment, the generated face model is accurately mapped to the previously established scene model using the position information (such as coordinates, orientation, etc.) of the face image in the depth image. Through coordinate transformation and model fusion technology, the device ensures that the face model is seamlessly connected with other parts of the scene model to form a complete scene model containing personalized face information.

[0054] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 2 , Figure 2 This is a flow chart of the second embodiment of the audio and video scene reconstruction method of the present application. Figure 2 As shown, the method may further include steps B10 to B30: Step B10, identifying a torso image corresponding to the face image in the scene depth image; In this embodiment, after acquiring the scene depth image and successfully identifying the face image, the device further uses deep learning and image segmentation technology to identify the torso image corresponding to the face image. This process involves understanding the human body structure and accurately segmenting the human body parts in the image. The device analyzes the distance information and light and shadow information in the depth image to determine the torso parts around the face image that belong to the same person.

[0055] Step B20, tracking the torso features of the torso image in real time; In this embodiment, once the torso image is recognized, the device uses a tracking algorithm (such as feature point-based tracking, optical flow method, or deep learning tracker) to track key torso features in the torso image in real time, including but not limited to the shape, position, and dynamic changes of parts such as shoulders, chest, waist, and limbs.

[0056] Step B30, mapping the torso features to a preset torso basic model to obtain a torso model corresponding to the face image.

[0057] In this embodiment, the device maps the tracked torso features to a preset torso basic model with a general human body structure. Preferably, in order to save storage and computing power, the image texture of the torso does not need to be mapped to the torso model, and only the preset torso model is used to record the participant's movements in real time. Finally, the device will obtain a torso model corresponding to the face image, with personalized features and dynamic changes.

[0058] As an example, specifically, after modeling the face, in order to improve the subsequent viewing experience, the human torso needs to be modeled. Because the torso is different from the face, the facial texture usually includes a lot of information, such as skin features, muscle direction details, etc., which can provide some help in identifying the facial expression or identity. However, the torso texture includes less information, and the torso mainly includes contour and action information. Therefore, only the torso needs to be modeled by tracking feature points, and there is no need to map the torso texture to the model. This method can significantly reduce the storage size of the model and reduce the computing resources consumed by modeling.

[0059] Based on the first embodiment and / or the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those of the above-mentioned embodiment 1 and / or the second embodiment can refer to the above introduction, and will not be repeated later. Figure 3, Figure 3 This is a flow chart of the third embodiment of the audio and video scene reconstruction method of the present application. Figure 3 As shown, the method may further include steps C10 to C20: Step C10, acquiring a sound signal through a microphone array; In this embodiment, the device is equipped with a microphone array, which consists of multiple microphones that work together to capture sound signals in the surrounding environment. The microphone array uses beamforming technology to enhance sound signals from a specific direction while suppressing environmental noise from other directions. In this way, the device can obtain clear and high-quality sound signals, providing a basis for speech recognition in subsequent steps.

[0060] Step C20, identifying a voice signal in the sound signal, and converting the voice signal into text information.

[0061] In this embodiment, after acquiring the sound signal, the device will use the speech recognition algorithm to recognize the speech signal therein, including the steps of preprocessing the speech signal (such as denoising, filtering and feature extraction), matching the acoustic model and decoding the language model. Finally, the device will convert the recognized speech signal into corresponding text information.

[0062] By acquiring sound signals through a microphone array and converting them into text information using a speech recognition algorithm, the device can provide users with a more intelligent and convenient interactive experience and make meeting records clearer and more accurate.

[0063] Furthermore, in a feasible implementation manner, after the above step C20, the method may further include steps C30 to C60: Step C30, identifying the sound source location of the sound signal in the target scene by using the microphone array; In this embodiment, the device uses the beamforming and time difference positioning technology of the microphone array to further analyze the sound signal to determine the specific position of the sound source in the target scene. This step relies on the time difference and intensity difference of the sound signal received by each microphone in the microphone array, and the three-dimensional coordinates of the sound source or the relative position relative to the microphone array are obtained by calculation.

[0064] Step C40, determining a target face image in each of the face images according to the sound source localization; In this embodiment, after obtaining the sound source location, the device will combine the previously recognized face image and determine the target face image associated with the sound signal by comparing the sound source position with the position of the face image in the scene.

[0065] Step C50, determining a display position of a text box according to the face position of the target face image in the complete scene model, wherein a distance between the display position and the face position is less than a preset threshold; In this embodiment, after determining the target facial image, the device will search for the three-dimensional position corresponding to the facial image in the complete scene model, and then determine the display position of the text box in the scene based on preset rules or algorithms (such as maintaining the relative position relationship between the text box and the face, avoiding blocking the face, etc.). This position should meet the condition that the distance between the position of the face is less than a preset threshold to ensure that the text information can be clearly displayed near the target face.

[0066] Step C60, displaying the text information in the text box.

[0067] In this embodiment, finally, the device displays the recognized text information in a determined text box, and may also include optimization of visual parameters such as format adjustment, color selection, and font size of the text information to ensure the readability and aesthetics of the text information in the scene.

[0068] The above steps further enhance the interactivity and practicality of the method. Through the steps of sound source localization, target face image determination, text box display position calculation and text information display, users can understand the speeches of people in the meeting more clearly.

[0069] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the audio and video scene reconstruction method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0070] This application also provides an audio and video scene reconstruction device, please refer to Figure 4 , the audio and video scene reconstruction device comprises: An image acquisition module 10 is used to acquire a scene depth image of a target scene through a camera array; A scene modeling module 20, configured to establish a scene model based on the scene depth image; A face recognition module 30, used to recognize a face image in the scene depth image and extract face features in the face image; A face modeling module 40, used for fitting the face features with a preset face model to obtain a face model corresponding to the face image; The scene fusion module 50 is used to map the face model to the scene model based on the position information of the face image in the scene depth image to obtain a complete scene model.

[0071] Optionally, the scene modeling module 20 is further used for: Traversing each of the scene depth images, and aligning the scene depth images to a preset reference coordinate system; Convert each pixel in the scene depth image into a point in the reference coordinate system to obtain a three-dimensional point cloud composed of each scene depth image; Meshing the three-dimensional point cloud to obtain a mesh model; The scene depth image is mapped to the grid model to obtain scene modeling.

[0072] Optionally, the audio and video scene reconstruction device is also used for: Tracking each of the facial features in each frame image after the scene depth image; The face model is updated in real time according to the facial features of the tracked moral.

[0073] Optionally, the audio and video scene reconstruction device is also used for: Identifying a torso image corresponding to the face image in the scene depth image; Tracking torso features of the torso image in real time; The torso features are mapped to a preset torso basic model to obtain a torso model corresponding to the face image.

[0074] Optionally, the audio and video scene reconstruction device is also used for: Acquire sound signals through a microphone array; A voice signal in the sound signal is recognized, and the voice signal is converted into text information.

[0075] Optionally, the audio and video scene reconstruction device is also used for: Identifying the sound source location of the sound signal in the target scene by using the microphone array; Determining a target face image in each of the face images according to the sound source localization; Determine a display position of a text box according to the face position of the target face image in the complete scene model, wherein the distance between the display position and the face position is less than a preset threshold; The text information is displayed in the text box.

[0076] The audio and video scene reconstruction device provided by the present application adopts the audio and video scene reconstruction method in the above embodiment, which can solve the technical problem of how to achieve complete and accurate recording of the conference scene. Compared with the prior art, the beneficial effects of the audio and video scene reconstruction device provided by the present application are the same as the beneficial effects of the audio and video scene reconstruction method provided by the above embodiment, and the other technical features in the audio and video scene reconstruction device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0077] The present application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the audio and video scene reconstruction method in the above-mentioned embodiment 1.

[0078] Reference below Figure 5 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic devices in the embodiments of the present application may include but are not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions: tablet computers), PMPs (Portable Media Players: portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0079] like Figure 5 As shown, the electronic device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the electronic device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows an electronic device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0080] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0081] The electronic device provided by the present application adopts the audio and video scene reconstruction method in the above embodiment, which can solve the technical problem of how to achieve complete and accurate recording of the conference scene. Compared with the prior art, the beneficial effects of the electronic device provided by the present application are the same as the beneficial effects of the audio and video scene reconstruction method provided by the above embodiment, and the other technical features in the electronic device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0082] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0083] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0084] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the audio and video scene reconstruction method in the above-mentioned embodiment.

[0085] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0086] The computer-readable storage medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0087] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by an electronic device, the electronic device: obtains a scene depth image of the target scene through a camera array; establishes a scene model based on the scene depth image; identifies a face image in the scene depth image, and extracts face features in the face image; fits the face features with a preset facial model to obtain a face model corresponding to the face image; and maps the face model to the scene model based on the position information of the face image in the scene depth image to obtain a complete scene model.

[0088] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0089] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0090] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0091] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned audio and video scene reconstruction method, and can solve the technical problem of how to achieve complete and accurate recording of conference scenes. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the audio and video scene reconstruction method provided in the above-mentioned embodiment, and will not be repeated here.

[0092] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned audio and video scene reconstruction method when executed by a processor.

[0093] The computer program product provided by this application can solve the technical problem of how to achieve complete and accurate recording of conference scenes. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the audio and video scene reconstruction method provided by the above embodiment, which will not be repeated here.

[0094] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for reconstructing an audio and video scene, characterized in that: The method comprises: Acquire a scene depth image of the target scene through a camera array; Establishing a scene model based on the scene depth image; Identifying a face image in the scene depth image, and extracting face features in the face image; Fitting the facial features to a preset facial model to obtain a facial model corresponding to the facial image; Based on the position information of the face image in the scene depth image, the face model is mapped to the scene model to obtain a complete scene model.

2. The audio and video scene reconstruction method according to claim 1, characterized in that: The step of establishing a scene model based on the scene depth image comprises: Traversing each of the scene depth images, and aligning the scene depth images to a preset reference coordinate system; Convert each pixel in the scene depth image into a point in the reference coordinate system to obtain a three-dimensional point cloud composed of each scene depth image; Meshing the three-dimensional point cloud to obtain a mesh model; The scene depth image is mapped to the grid model to obtain scene modeling.

3. The audio and video scene reconstruction method according to claim 1, characterized in that: After the step of fitting the facial features with a preset facial model to obtain a facial model corresponding to the facial image, the method further includes: Tracking each of the facial features in each frame image after the scene depth image; The face model is updated in real time according to each of the tracked face features.

4. The audio and video scene reconstruction method according to claim 1, characterized in that: The method further comprises: Identifying a torso image corresponding to the face image in the scene depth image; Tracking torso features of the torso image in real time; The torso features are mapped to a preset torso basic model to obtain a torso model corresponding to the face image.

5. The audio and video scene reconstruction method according to claim 1, characterized in that: The method further comprises: Acquire sound signals through a microphone array; A voice signal in the sound signal is recognized, and the voice signal is converted into text information.

6. The audio and video scene reconstruction method according to claim 5, characterized in that: After the step of converting the voice signal into text information, the method further includes: Identifying the sound source location of the sound signal in the target scene by using the microphone array; Determining a target face image in each of the face images according to the sound source localization; Determine a display position of a text box according to the face position of the target face image in the complete scene model, wherein the distance between the display position and the face position is less than a preset threshold; The text information is displayed in the text box.

7. An audio and video scene reconstruction device, characterized in that: The device comprises: An image acquisition module, used to acquire a scene depth image of a target scene through a camera array; A scene modeling module, used to establish a scene model based on the scene depth image; A face recognition module, used to recognize a face image in the scene depth image and extract face features in the face image; A face modeling module, used for fitting the face features with a preset face model to obtain a face model corresponding to the face image; A scene fusion module is used to map the face model to the scene model based on the position information of the face image in the scene depth image to obtain a complete scene model.

8. An electronic device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the audio and video scene reconstruction method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the audio and video scene reconstruction method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the audio and video scene reconstruction method according to any one of claims 1 to 6 are implemented.

Images