Multi-camera module overlapping view field configuration and close-up picture acquisition method and device

Abstract

The invention relates to a multi-camera module overlapping view field configuration and close-up picture acquisition method and device, and belongs to the technical field of video signal processing. Comprising the following steps: sequentially arranging a plurality of camera modules along a preset direction; obtaining a decoding stream of each camera module, and performing human shape detection by using the decoding stream to obtain candidate person image coordinates corresponding to each decoding stream; performing detection target de-duplication processing on all image coordinates; according to an audio and video angle mapping table, converting the de-duplicated image coordinates into corresponding image angles to obtain an image angle set; according to a matching rule, comparing the audio angle with the image angle set so as to determine a target object angle; calculating the boundary distance of the target object angle in the view field of each camera module, and screening the maximum boundary distance; and extracting an output picture corresponding to the maximum boundary distance so as to obtain a close-up picture. According to the invention, high-definition seamless close-up picture acquisition is realized, and the problem that panoramic coverage and high-definition close-up are difficult to consider at the same time is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video signal processing, in particular to a multi-camera module overlapping field of view configuration and close-up picture acquisition method and device. BACKGROUND

[0002] In the field of visual acquisition in a conference scene, a panoramic device is a core device for realizing 360° all-around picture coverage. Its technical solution mainly falls into two categories: one is to use multiple camera modules to splice the field of view to construct a panoramic view, and the other is to directly obtain a panoramic image by using a panoramic lens (such as a super-wide-angle fisheye lens) once. However, in actual application, both of the two mainstream solutions have the technical pain point that it is difficult to balance panoramic coverage and high-definition close-up output, especially when a speaker and other target objects need to be clearly presented in a close-up, the problem is more prominent.

[0003] For the multi-camera module splicing solution, the existing technology generally designs a small overlapping angle of the fields of view of adjacent camera modules in order to reduce picture redundancy and improve data processing efficiency. Although this design can reduce the data amount of the repeated area when splicing a panoramic view, when a speaker and other targets that need to be presented in a close-up are located in the field-of-view junction area of two camera modules, if a high-definition close-up picture is to be output, usually only two processing methods can be used: one is cross-border splicing, that is, the picture of the junction area collected by the two modules is spliced and fused by an algorithm, and then a close-up is taken, which is easy to cause obvious seams in the close-up picture due to the imaging parameter difference of the two modules and the edge processing precision limitation of the splicing algorithm, and the picture distortion degree is uneven, which seriously affects the visual effect; the other is panoramic cutting, that is, the target area is directly cut from the complete 360° panoramic picture as a close-up, since the resolution of the panoramic picture needs to be evenly distributed to the entire 360° field of view, the number of pixels occupied by a single target area is limited, and the close-up picture after cutting often has the problem of insufficient resolution, which cannot clearly present the details such as the facial texture of a person.

[0004] And the scheme of using ultra-wide-angle fisheye lens to obtain panorama generally faces the core problem of low effective pixel utilization rate. The typical fisheye lens is affected by the optical structure characteristics, and its spatial resolution presents a distribution law of gradual attenuation from the optical axis center to the edge: the pixel density of the center area of the lens is high, and the imaging quality is good, but this area in the conference scene often only corresponds to the empty ceiling or ground, which is not the key information collection area; and the key area where the participants are located is usually in the annular band deviating from the optical axis center of the lens, which is just in the resolution attenuation area of the lens image field, and the basic imaging clarity is low. In the subsequent image processing process, the image of the annular band area needs to be remapped to a rectangular panoramic picture conforming to the habit of human visual perception, which will introduce a certain image distortion; if a close-up picture of the participant is further cropped and enlarged from the rectangular panoramic picture, interpolation algorithm is also needed to supplement the pixels, and the interpolation and enlargement process will not only further lose the picture details, but also superimpose the errors introduced by the early distortion correction, finally leading to the general clarity of the panoramic picture, and the clarity problem of the close-up picture is particularly prominent, the high-frequency details such as the texture of the face of the person are seriously blurred, and there is a significant gap between the actual picture quality perceived by the user and the nominal theoretical sensor pixel specification of the device, forming a significant experience gap.

[0005] In summary, the technical scheme of the existing conference panoramic device does not fully consider the demand for high-definition close-up without seams in the hardware field of view layout stage, lacks an overall processing method that can reserve redundant center image resources for this demand, and the contradiction between panoramic coverage and high-definition close-up cannot be effectively solved. SUMMARY

[0006] Therefore, the technical problem to be solved by the present application is to overcome the problems of seams, distortion, insufficient resolution, low effective pixel utilization rate and loss of close-up picture details in the prior art when outputting a close-up picture in the field of view junction area.

[0007] In a first aspect, to solve the above technical problems, the present application provides a multi-camera module overlapping field of view configuration and close-up picture acquisition method, comprising: S1, sequentially arranging a plurality of camera modules along a preset direction and a preset field of view overlap area; obtaining a decoding stream output by each camera module, performing human body detection on the decoding stream respectively to obtain candidate human image coordinates corresponding to each decoding stream; and performing detection target deduplication processing on all candidate human image coordinates; S2, converting the deduplicated candidate human image coordinates into corresponding image angles according to an audio-video angle mapping table to obtain an image angle set; S3, obtaining an audio angle; comparing the audio angle with the image angle set according to a preset matching rule to obtain a matching combination; and determining a target object angle according to the matching combination; S4, calculate a boundary distance of the target object angle in each field of view of the camera module, to obtain a boundary distance set; S5, screen out a maximum value in the boundary distance set, and obtain a matching camera module according to the maximum value; extract an output picture of the matching camera module, and obtain a close-up picture of the target object according to the output picture.

[0008] In an embodiment of the present application, in the S3, the matching rule is: If no sound source signal is detected as human voice or no human-shaped target is detected in the current frame, the current frame is directly skipped, and it is determined that there is no matching combination; if there is human voice and human-shaped detection result in the current frame, an image angle closest to the audio angle distance in the image angle set is selected as the matching combination.

[0009] In an embodiment of the present application, in the S4, the step of calculating the boundary distance of the target object angle in each field of view of the camera module is: obtaining a starting angle of the field of view of the camera module; calculating a difference value between the target object angle and the starting angle of the field of view of the camera module to obtain a first difference value; obtaining an ending angle of the field of view of the camera module; calculating a difference value between the ending angle of the field of view of the camera module and the target object angle to obtain a second difference value; comparing the first difference value and the second difference value, and selecting a minimum value as the boundary distance.

[0010] In an embodiment of the present application, in the S5, the process of extracting the output picture of the matching camera module and obtaining the close-up picture of the target object according to the output picture includes: cropping a region where the target object is located from the output picture to obtain a cropped picture; performing distortion correction on the cropped picture by using overlap distortion correction, and performing smoothing processing on an edge region of the cropped picture by using a center weight map to obtain the close-up picture of the target object.

[0011] In an embodiment of the present application, the S5 further includes a step of splicing processing the close-up picture and a non-close-up picture; wherein the step of splicing processing is: limiting a splicing range of a panoramic field of view, and performing splicing processing on camera module output pictures in the splicing range; setting a splicing seam of the close-up picture and a plurality of non-close-up pictures in a preset low-attention region.

[0012] In an embodiment of the present application, in the S1, the preset field of view overlap region is 60°.

[0013] In a second aspect, to solve the above technical problems, the present application provides a multi-camera module overlapping field of view configuration and close-up picture acquisition device, which is used to implement the above method, and comprises: a camera module array, comprising a plurality of camera modules, used to provide a plurality of original images and output corresponding decoding streams according to each original image; a human shape detection module, used to acquire the decoding streams output by each camera module, perform human shape detection on the decoding streams respectively to obtain candidate human image coordinates corresponding to each decoding stream, and perform detection target deduplication processing on all candidate human image coordinates; an image angle acquisition module, used to convert the deduplicated candidate human image coordinates into corresponding image angles according to an audio-video angle mapping table to obtain an image angle set; an audio positioning module, used to acquire an audio angle, compare the audio angle with the image angle set according to a matching rule to obtain a matching combination, and determine a target object angle according to the matching combination; a calculation module, used to calculate the boundary distance of the target object angle in the field of view of each camera module to obtain a boundary distance set; a close-up picture output module, used to screen out the maximum value in the boundary distance set, obtain a matching camera module according to the maximum value, extract the output picture of the matching camera module, and obtain a close-up picture of a target object according to the output picture.

[0014] In an embodiment of the present application, the plurality of camera modules are sequentially arranged along a preset annular direction.

[0015] In an embodiment of the present application, the camera module array comprises a first camera module, a second camera module, a third camera module and a fourth camera module, the field of view coverage range of the first camera module is [30°, 150°), the field of view coverage range of the second camera module is [90°, 210°), the field of view coverage range of the third camera module is [150°, 270°), and the field of view coverage range of the fourth camera module is [210°, 330°).

[0016] In an embodiment of the present application, the audio positioning module comprises a microphone array, and the microphone array comprises a circular ring and a plurality of microphones.

[0017] The above technical solution of the present application has the following beneficial effects compared with the prior art: (1) The multi-camera module overlapping field of view configuration and close-up picture acquisition method and device effectively solves the problem that panoramic coverage and high-definition close-up are difficult to be considered, and realizes the acquisition of high-definition seamless close-up picture. The multiple camera modules are sequentially arranged along the preset direction and the overlapping area, ensuring panoramic coverage and providing sufficient overlapping area for subsequent processing. The decoded stream output by each camera module is subjected to human body detection to obtain candidate human image coordinates and remove duplicates, avoiding repeated detection and improving the accuracy and efficiency of target positioning. The audio-video angle mapping table is used to convert the de-duplicated candidate human image coordinates into an image angle set. At the same time, the audio angle is obtained and compared with the image angle set to obtain a matching combination. This audio-video cooperative processing mechanism improves the accuracy of target object positioning and the quality of close-up picture through sound source direction pre-screening. The boundary distance of the target object angle in the field of view of each camera module is calculated, and the maximum value is selected to determine the matching camera module. The output picture of the module is extracted to obtain the close-up picture of the target object. This process ensures the clarity and stability of the close-up picture and avoids the problem of cross-module distortion splicing, thereby improving the quality of the close-up picture.

[0018] (2) The present application adopts a ring layout of not less than 2 camera modules, combines the optional panoramic angle range and the logic of expanding the adjacent overlapping angle, ensures panoramic coverage and provides sufficient overlapping area, and is beneficial to smooth image splicing. The introduction of the cooperative constraint of overlapping angle and boundary distance threshold value ensures that the boundary person is always near the center picture area of the adjacent camera module, avoids image distortion and splicing problems, and improves the picture quality. The panoramic field of view is cut by a hardware and software joint strategy, which realizes efficient cutting through hardware optimization and software algorithm adjustment, ensures the resolution and quality of the cut picture, and meets the needs of different scenes. The microphone array can quickly locate and prioritize the speaker close-up lens. Through accurate sound source positioning and voice activity detection, the device can quickly identify the speaker and sort them by position and importance, ensuring the accuracy and timeliness of the close-up picture. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to make the content of the present application more easily understood, the present application will be further described in detail below according to the specific embodiments of the present application and in conjunction with the accompanying drawings.

[0020] Figure 1 A flow chart of a multi-camera module overlapping field of view configuration and close-up picture acquisition method in a preferred embodiment of the present application; Figure 2 A schematic diagram of a camera module imaging area flattened into a 50% overlapping rectangular picture in a preferred embodiment of the present application; Figure 3 A schematic diagram of two camera module picture overlapping pictures in a preferred embodiment of the present application; Figure 4A multi-camera module overlapping field of view configuration and close-up picture acquisition device structure diagram in a preferred embodiment of the present application; Figure 5 A four camera module field of view range schematic diagram in a preferred embodiment of the present application; Figure 6 An audio positioning module principle schematic diagram in a preferred embodiment of the present application; Figure 7 A panoramic stitching schematic diagram in a preferred embodiment of the present application; Figure 8 A multi-camera module overlapping field of view configuration and close-up picture acquisition device execution flowchart in a preferred embodiment of the present application.

[0021] Description of the figures: 1, camera module; 11, camera; 2, microphone array; 21, microphone. DETAILED DESCRIPTION

[0022] The present application will be further described below in conjunction with the drawings and specific embodiments, so that those skilled in the art can better understand the present application and implement it, but the embodiments are not limiting the present application.

[0023] Embodiment one: Referring to Figure 1 The embodiment of the present application provides a multi-camera module overlapping field of view configuration and close-up picture acquisition method, including but not limited to the following steps: S1, sequentially arrange a plurality of camera modules along a preset direction and a preset field of view overlap area; obtain the decoding stream output by each camera module, use the decoding stream to perform human detection respectively, and obtain the candidate human image coordinates corresponding to each decoding stream; perform detection target deduplication processing on all candidate human image coordinates; S2, according to the audio-video angle mapping table, convert the deduplicated candidate human image coordinates into corresponding image angles, and obtain an image angle set; S3, obtain an audio angle; according to a preset matching rule, compare the audio angle with the image angle set, and obtain a matching combination; according to the matching combination, determine a target object angle; S4, calculate the boundary distance of the target object angle in each camera module field of view, and obtain a boundary distance set; S5, filter out the maximum value in the boundary distance set, and according to the maximum value, obtain a matching camera module; extract the output picture of the matching camera module, and according to the output picture, obtain a close-up picture of the target object.

[0024] The multi-camera module overlapping field of view configuration and close-up picture acquisition method of the embodiment of the present application realizes high-definition seamless close-up picture acquisition, and solves the problem that panoramic coverage and high-definition close-up are difficult to be considered together. The multiple camera modules are sequentially arranged along the preset direction and the preset field of view overlapping area, ensuring panoramic coverage while providing sufficient overlapping area for subsequent processing. The decoded stream output by each camera module is subjected to human body detection to obtain candidate human image coordinates corresponding to each decoded stream, and the detection target deduplication processing is performed. This process effectively avoids the repeated detection problem caused by the overlapping area, and improves the accuracy and efficiency of target positioning. The audio-video angle mapping table is used to convert the deduplicated candidate human image coordinates into corresponding image angles to obtain an image angle set. At the same time, the audio angle is obtained, and according to the preset matching rule, the audio angle is compared with the image angle set to obtain a matching combination. This audio-video cooperative processing mechanism reduces irrelevant picture switching through sound source direction pre-screening, and improves the accuracy of target object positioning and the quality of close-up pictures. The boundary distance of the target object angle in the field of view of each camera module is calculated to obtain a boundary distance set, and the maximum value in the boundary distance set is screened out to determine the matching camera module. The output picture of the matching camera module is extracted to obtain a close-up picture of the target object. This process ensures the clarity and stability of the close-up picture and avoids the cross-module distortion splicing problem, significantly improving the quality of the close-up picture. The method of the embodiment of the present application overcomes the problems commonly encountered in the prior art when outputting a close-up picture at the field of view junction area, such as seams, distortion, insufficient resolution, low effective pixel utilization, and loss of close-up picture details. Through the above design, high-definition seamless close-up picture acquisition is realized, and the problem that panoramic coverage and high-definition close-up are difficult to be considered together is solved. In addition, the method improves the user experience and is suitable for scenes such as conferences, teaching, monitoring, and the like that require high-quality close-up pictures.

[0025] Specifically, in step S1, multiple camera modules are sequentially arranged along a preset direction and a preset field of view overlapping area. In the embodiment of the present application, the number of camera modules is preferably 4, and each camera module is provided with a camera for image acquisition. For convenience of description, the cameras of the four camera modules are defined as a first camera, a second camera, a third camera, and a fourth camera.

[0026] Further, in the embodiments of the present application, the preset field of view overlap region is set to 60°, for example, the field of view overlap region of the first camera module and the second camera module is specifically [90°, 150°). The relationship that needs to be met for the overlap region is: the overlap region angle ≥ a preset threshold (such as 45°-60°). Such a design ensures that any person located at the edge ± δ (such as ± 10°) of a module is in the center weight region of another module (for example, the center weight region is defined as the middle 40°-60° of the field of view of the camera module). In order to facilitate understanding, the imaging area of the camera module is flattened into a 50% overlapping rectangular picture, as shown in Figure 2 .

[0027] Further, in Figure 2 , ola1, ola2, ola3 respectively represent the picture overlap region of the first camera and the second camera, the second camera and the third camera, and the third camera and the fourth camera. center_1, center_2, center_3 and center_4 respectively represent the center position of the corresponding camera picture. δ0 and δ2 represent two boundaries of the first camera, δ1 and δ3 represent two boundaries of the second camera, δ2 and δ4 represent two boundaries of the third camera, and δ3 and δ5 represent two boundaries of the fourth camera. After the pictures of the four cameras are spliced together, a panoramic display picture is formed. Among them, the boundaries δ0 and δ5 correspond to people close to the edge of the panoramic picture, and there is no need to splice. For the person near the boundary δ2 of the first camera, he is also near the center position of the picture of the second camera, so when the person at this position needs to be presented in a close-up picture, only the picture needs to be directly cropped from the picture of the second camera, without the need for splicing, thereby directly avoiding the splicing problem of the close-up picture. Similarly, the same processing is performed on the speakers at positions δ1, δ3 and δ4. The speaker at position δ5 is located at the edge position on the other side of the panoramic picture, and the picture can be directly cropped from the fourth camera.

[0028] It should be noted that the number of camera modules and the specific way of arranging them in sequence along the preset direction and the preset field of view overlap area can be flexibly set according to actual application requirements. For example, if the preset field of view overlap area is set to 50°, the effective field of view of each camera module can be flexibly adjusted between 50° and 60° through a software dynamic cropping function. This adjustment method can ensure the resolution of the picture while reasonably controlling the field of view redundancy, achieving an optimized balance between the field of view range and the picture quality. When increased to 5 camera modules, the physical field of view angle of each camera module is 110°, and through logical splicing, a 330° panoramic coverage can be achieved, and an overlap area of 45° is maintained between adjacent camera modules. In this configuration, other related algorithms remain consistent with the configuration of 4 camera modules. When increased to 6 camera modules, the physical field of view angle of each camera module is 120°, and through logical splicing, a 360° panoramic coverage can be achieved, and an overlap area of 60° is maintained between adjacent camera modules. Similarly, in this configuration, other related algorithms also remain consistent.

[0029] Further, in step S1, the decoding stream (such as YUV code stream) output by each camera module is obtained, and human body detection is performed on the decoding stream to obtain candidate human image coordinates corresponding to each decoding stream. The specific steps of detecting target deduplication processing for all candidate human image coordinates are as follows: S110, human body detection is performed on the decoding stream, and AI face / human body detection algorithm is used to obtain candidate human image coordinates corresponding to each decoding stream. The AI face / human body detection algorithm, such as YOLOv8, SSD, Faster R-CNN, etc., can efficiently recognize and locate human body through deep learning technology, and provide accurate coordinate information for subsequent processing.

[0030] This step relies on the field of view coverage range of each module to ensure that the human targets in the conference scene can be captured by at least one module, especially through the design of a 60° overlap area, so that the characters in the boundary area can be detected by two adjacent modules at the same time, providing redundant data for subsequent deduplication and optimization.

[0031] S120, detecting target deduplication processing is performed on all candidate human image coordinates.

[0032] Exemplarily, the candidate human image coordinates output by all camera modules are aggregated and analyzed, and whether it is the same human target is judged by calculating the intersection over union (IoU) of the bounding boxes of the detection results of different modules (such as IoU≥0.7 is judged as a repeated target). For repeated targets, the coordinate data with clearer imaging is retained (which can be determined by evaluating the sharpness of the pixels in the bounding box and the completeness of the target), and finally a set of unique candidate human image coordinates after deduplication is obtained, avoiding the same person being matched multiple times in subsequent processing.

[0033] Specifically, in step S2, the image angle set {θ_visual} is obtained based on the pre-calibrated audio-video angle mapping table. For the construction of the audio-video angle mapping table, an exemplary description is given as follows.

[0034] Exemplarily, when estimating the sound source direction using the microphone array, 360° in the horizontal direction is uniformly divided into several equal parts, such as 30 parts, each covering 12°. At the same time, the horizontal space corresponding to the image coordinates is also uniformly divided into 30 parts according to the same number of parts. In the calibration stage, when the target object speaks in different regions, the audio angle and the corresponding image coordinates are recorded respectively. Based on these data, an audio-video angle mapping table between the audio angle and the image coordinates is established. Generally, this mapping is one-to-one, that is, a specific image coordinate region corresponds to a specific sound source angle. In the actual processing stage, when the image coordinate region of a certain speaker is detected, according to the audio-video angle mapping table, this region may correspond to one or more audio angle ranges. The closer the speaker is to the camera, the larger the image region he detects, and the wider the corresponding audio angle interval. If these audio angle ranges match the audio angle θ_audio obtained by sound source direction finding, the multi-modal confirmation of the speaker who needs to be close-up can be completed.

[0035] Specifically, in step S3, the specific steps for determining the angle of the target object are: S310, obtaining the audio angle θ_audio, the specific steps are: S311, audio acquisition and voice judgment. Audio data is collected using multiple microphones (mic). Based on the audio data, short-time energy detection, zero-crossing rate detection, Gaussian mixture model detection, AI neural network model, etc. are used to judge whether the current frame is human voice. Among them, short-time energy detection identifies human voice by analyzing the energy change of the audio signal; zero-crossing rate detection judges according to the zero-crossing rate characteristics of the audio signal; Gaussian mixture model detection classifies audio features using a statistical model. And the AI neural network model, especially the voice activity detection (VAD) algorithm based on deep learning, can more accurately identify whether the current frame contains human voice by learning a large amount of spectral features, energy changes and other key information of audio data, thereby providing accurate input basis for subsequent audio angle calculation.

[0036] S312, audio angle θ_audio is obtained by processing the collected audio data by using methods such as Generalized Cross-Correlation with Phase Transform (GCC-PHAT) and Steered Response Power (SRP) beamforming.

[0037] S320, the audio angle θ_audio is compared with the image angle set {θ_visual} according to a preset matching rule to determine whether they match, so as to obtain a matching combination. The matching rule is specifically: if no sound source signal is detected as human voice in the current frame, or no human-shaped target is detected, the frame is directly skipped, and no matching combination is determined; if the sound source signal is detected as human voice in the current frame, and there is a human-shaped detection result, the image angle closest to the audio angle θ_audio is selected from the image angle set {θ_visual}. The closest image angle and the audio angle θ_audio are combined as a matching combination.

[0038] Further, each image angle in the image angle set {θ_visual} is compared with the audio angle θ_audio in sequence to determine whether the above matching rule is met. If yes, a matching combination is successfully obtained; if not, the next frame of data is processed until a matching combination meeting the condition is found or the processing of all frames is completed.

[0039] S330, the target object angle θ_close-up is finally determined according to the obtained matching combination.

[0040] Specifically, in step S4, the step of calculating the boundary distance of the target object angle in the field of view of each camera module is: S410, the start angle of the field of view of the camera module is obtained; the difference between the target object angle and the start angle of the field of view of the camera module is calculated to obtain a first difference; S420, the end angle of the field of view of the camera module is obtained; the difference between the end angle of the field of view of the camera module and the target object angle is calculated to obtain a second difference; S430, the first difference and the second difference are compared, and the minimum value is selected as the boundary distance . The expression of the boundary distance is: ; wherein, θ_close-up represents the target object angle, θ_start represents the start angle of the first camera module, θ_end represents the end angle of the first camera module, and θ_audio represents the audio angle. End angle of camera module, represents the minimum value function. The boundary distance set The expression is: represents the total number of camera modules,

[0041] Specifically, referring to FIG. 5, in step S5, the process of extracting the output picture of the matching camera module and obtaining the close-up picture of the target object according to the output picture includes: Figure 3 cropping the region where the target object is located from the output picture to obtain a cropped picture; performing distortion correction on the cropped picture using overlap distortion correction, and performing smoothing processing on the edge region of the cropped picture using a center weight map. Further,

[0042] Two camera module picture overlapping configurations are shown. Among them, θ0 and θ2 respectively correspond to the left boundary and right boundary of the picture of the first camera module, θ1 and θ3 respectively correspond to the left boundary and right boundary of the picture of the second camera module, center_1 and center_2 respectively represent the picture center positions of the first and second camera modules. θ is the example angle position of the actor, which is located near the left boundary of the picture of the second camera module. Its boundary distance in the second camera module is Δ2 = min(θ-θ1, θ3-θ), and similarly, its boundary distance in the first camera module is Δ1 = min(θ-θ0, θ2-θ). As can be seen from the figure, although the actor is located at the left boundary position of the picture of the second camera module, it is also near the picture center position (center_1) of the first camera module. Therefore, the picture of the first camera module can be directly selected for close-up cropping processing. In this way, not only can the splicing processing be effectively avoided, but also the close-up splicing seam problem caused by splicing can be fundamentally eliminated, thereby improving the overall quality and visual effect of the picture. Figure 3 Specifically, after step S5, there is also a step of splicing the close-up picture with the non-close-up picture, that is, panoramic splicing processing; wherein the splicing processing step is:

[0043] S610, limit the splicing range of the panoramic field of view, and splice the output picture of the camera module in the splicing range. In the embodiment of the present application, the optimal splicing range is 300° (the global azimuth angle is θ ∈ [0°, 360°)), and the intervals [0°, 30°) and [330°, 360°) are discarded. This optimization aims to reduce the amount of calculation and bandwidth consumption, while ensuring the integrity and practicality of the spliced picture.

[0044] ​​​S620, set the splicing seam position, set the splicing seam of the close-up picture and multiple non-close-up pictures in the preset low attention area. For example, in a conference room scene, the splicing seam can be set in the direction of the whiteboard large screen and the like, which is not easy to be focused by the audience. In this way, the influence of the splicing seam on the visual experience can be effectively reduced, and the natural transition and overall aesthetics of the panoramic picture are ensured.

[0045] The embodiment of the application ensures that the person located at the boundary of a certain camera module is simultaneously present in the center region of the adjacent camera module picture by setting a larger overlap angle, effectively solving the splicing seam problem of the close-up picture. In addition, the single-module center region is selected for close-up, avoiding cross-module distortion splicing and improving the clarity of the close-up picture. At the same time, the method cooperates with audio and video processing, uses sound source direction pre-screening, reduces irrelevant picture switching, and thus improves the speaker close-up split screen effect.

[0046] Embodiment two: Based on the same inventive concept, the embodiment provides a multi-camera module overlapping field of view configuration and close-up picture acquisition device, which has a similar problem solving principle to the multi-camera module overlapping field of view configuration and close-up picture acquisition method provided in embodiment one, and the repeated parts will not be described again.

[0047] Referring to Figure 4 The embodiment provides a multi-camera module overlapping field of view configuration and close-up picture acquisition device for implementing the method described in embodiment one, which comprises: A camera module array comprising a plurality of camera modules 1 for providing a plurality of original images and outputting corresponding decoding streams according to each original image; A human form detection module for obtaining the decoding streams output by each camera module 1, performing human form detection on the decoding streams respectively to obtain candidate human image coordinates corresponding to each decoding stream, and performing detection target deduplication processing on all candidate human image coordinates; An image angle acquisition module for converting the deduplicated candidate human image coordinates into corresponding image angles according to an audio-video angle mapping table to obtain an image angle set; An audio positioning module for obtaining an audio angle, comparing the audio angle with the image angle set according to a matching rule to obtain a matching combination, and determining a target object angle according to the matching combination; A calculation module for calculating the boundary distance of the target object angle in the field of view of each camera module 1 to obtain a boundary distance set; A close-up picture output module for screening out the maximum value in the boundary distance set, obtaining a matching camera module 1 according to the maximum value, extracting the output picture of the matching camera module 1, and obtaining a close-up picture of the target object according to the output picture.

[0048] Specifically, in the present embodiment, the camera module array includes a plurality of camera modules 1; the plurality of camera modules 1 are sequentially arranged along a preset annular direction, and each includes a camera 11.

[0049] Exemplarily, the camera module array includes a first camera module, a second camera module, a third camera module and a fourth camera module; wherein the first camera module, the second camera module, the third camera module and the fourth camera module are sequentially arranged along a preset annular direction, and each includes a camera 11. In addition, in the present embodiment, the field of view angle of each camera module is about 120°. It should be noted that the design of the camera module array has certain flexibility and expansibility. In addition to the above four camera modules, according to the actual application requirements, the camera module array can further include a fifth camera module, a sixth camera module and more camera modules. For each newly added ith camera module, its field of view angle can be adjusted and set according to the specific application scene and monitoring requirements.

[0050] Specifically, referring to Figure 5 , the field of view coverage range of the first camera module is [30°, 150°), the field of view coverage range of the second camera module is [90°, 210°), the field of view coverage range of the third camera module is [150°, 270°), and the field of view coverage range of the fourth camera module is [210°, 330°). Wherein, the symbols del1 and del2 in the figure represent the discarded intervals, which are [0°, 30°) and [330°, 360°) respectively.

[0051] Specifically, referring to Figure 4 , the audio positioning module includes a microphone array 2, and the microphone array 2 includes a circular ring and a plurality of microphones 21. In the present embodiment, the microphone array 2 adopts 6 MEMS microphones in a ring layout. Specifically, 6 MEMS microphones of the same type are selected and uniformly arranged on a circular ring with a radius of 5-10 cm to form a uniform ring-shaped microphone array 2.

[0052] Further, the device described in the embodiment also includes a touch panel, a touch PBCA (touch electronic circuit board assembly), an indicator lamp cover, a first lens fixing member, a lens group, and a second lens fixing member. The touch panel allows touch operations. The touch PBCA, disposed below the touch panel, is used to process touch inputs and convert them into electronic signals. The indicator lamp cover, disposed on the side of the touch PBCA away from the touch panel (i.e., below the touch PBCA), is used not only to protect the internal indicator lamp but also to ensure that light can effectively convey state information. The first lens fixing member, disposed on the side of the indicator lamp cover away from the touch PBCA, provides stable support for the camera module 1. The lens group, disposed on the side of the first lens fixing member away from the indicator lamp cover, is used to install the camera module 1. The second lens fixing member, disposed on the side of the lens group away from the first lens fixing member, works in conjunction with the first lens fixing member to ensure accurate positioning and stability of the camera module 1. The above-mentioned components are assembled in a coaxial stacking manner, ensuring the compactness of the structure and the integration of the functions, thereby forming a highly integrated front-end interaction and imaging device, as shown in Figure 4 The design not only optimizes space utilization but also improves the overall performance and reliability of the device, allowing users to enjoy a smooth interaction experience and clear imaging effect.

[0053] Further, the audio positioning module also includes a sound source direction finding sub-module and a human voice activity detection sub-module. In the sound source direction finding sub-module, gcc-phat, srp, and other algorithms are used to accurately obtain the direction of the sound source. In the human voice activity detection sub-module, accurate judgment is made as to whether the current frame is human voice, as shown in Figure 6 In Figure 6 , X1, X2, X3, X4, X5, X6 represent the data collected by each microphone 21 on the uniform circular array. The sound source direction finding sub-module uses these data to find the direction of the sound source, obtaining the most likely sound source direction θ_audio∈[0°, 360°). The human voice activity detection sub-module (the detection method can refer to step S311 in Embodiment One) uses these data to detect human voice activity, outputting a decision label speech_flag for whether the current frame is speech. The result of speech_flag is 0 or 1. When speech_flag=0, it indicates a non-speech frame, and when speech_flag=1, it indicates a speech frame.

[0054] Exemplarily, according to the angle distribution of each camera module described above, the pictures of adjacent camera modules have an overlapping area of 60° (50%). When the speaker is located within the pictures of two camera modules, the camera module picture in which the speaker is near the center of the picture is preferentially selected and is subjected to non-splicing cropping processing, thereby serving as a candidate for the close-up picture. At the same time, the sound source direction finding submodule and the human voice activity detection submodule will respectively obtain the sound source direction and the human voice label, and these information will be used to accurately screen the speaker who is speaking. The device further combines the audio angle information, the human voice label, and the candidate close-up speaker image angle information to perform comprehensive information matching, thereby finally determining the close-up picture of the speaker. In addition, the device will also appropriately logically splice the pictures of all camera modules to completely present the close-up picture of the speaker, ensuring the coherence and integrity of the picture.

[0055] Further, the device described in the embodiment also includes a panoramic splicing module, which is specially used for efficiently splicing the close-up picture and the non-close-up picture. In the panoramic splicing module, picture fusion within a range of 300° is preferentially selected, which can not only significantly reduce the complexity of operation processing, but also effectively reduce the bandwidth resources required for data transmission, thereby improving the overall performance and operation efficiency of the device. In addition, in order to further optimize the splicing effect, the splicing seam is preferably arranged in a low attention area, such as the direction of the whiteboard large screen in the conference room, as shown in FIG. 8. Such a design can minimize the impact of the splicing seam on the visual experience of the audience, ensuring the natural transition and overall aesthetics of the picture. Figure 7

[0056] Further, the device described in the embodiment also includes a panoramic and close-up picture superimposed display module, which is used for the display mode of the panoramic picture and the close-up picture, and the switching processing of the close-up picture. Through this module, the user can simultaneously view the panoramic picture and the close-up picture, and obtain a more rich and intuitive visual experience.

[0057] ​Further, the device described in this embodiment also includes a calibration and data storage module. This module is used to store the key parameters and data of each camera module 1 to ensure the efficient operation of the device and the accuracy of data processing. Storing the key parameters and data of each camera module 1 specifically includes: the intrinsic parameters of the camera module 1, such as the optical characteristics inside the camera module 1, such as focal length, optical center position, etc.; the extrinsic parameters of the camera module 1, such as the position and direction of the camera module 1 in space; the overlap distortion correction LUT (lookup table), which is used to correct the distortion of the camera module 1 in the overlap area, to ensure the accuracy and consistency of the image when splicing; the center weight map, which is used to assign different weights to pixels in different areas during image processing to optimize image quality and splicing effect; the audio and video coordinate mapping function, which is used to map and align audio signals and video signals in spatial coordinates to ensure the synchronization and consistency of audio and video in space. Through the calibration and data storage module, the device can effectively manage and utilize these key data, thereby improving the precision and efficiency of video processing and providing solid data support for subsequent image splicing, close-up cropping, etc.

[0058] Further, referring to Figure 8 the overall processing flow of the device described in this embodiment is as follows: Step 1: Use the microphone array 2 to collect audio data, and the human voice activity detection sub-module and the sound source direction finding sub-module to perform human voice activity detection and sound source direction finding processing respectively. Through this process, the speech tag (speech_flag) and the sound source direction (θ_audio) are obtained. Among them, the sound source direction θ_audio represents the incident direction of the speaker relative to the microphone array 2.

[0059] Step 2: Based on the image data collected by the multi-camera module, image calibration is performed, and the calibrated and calibrated key data is stored for future use. The human body detection module uses the original image data and the calibration data to obtain the image angle set {θ_visual} of the speaker. These angles follow the mapping relationship between the human body coordinates established in the calibration process and the incident angle of the speaker sound source, which has the same physical meaning as θ_audio, both representing the incident direction of the speaker relative to the microphone array 2.

[0060] Step 3: The audio positioning module fuses the audio and video information. Through fusion processing, the angle of the speaker who needs to be close-up, i.e. the target object angle θ_close-up, is finally obtained.

[0061] Step 4: The close-up picture output module combines the target object angle θ_close-up and its human body image coordinate information to crop the close-up speaker's picture from a specific camera module and perform appropriate scaling and other processing to ensure the quality and visual effect of the close-up picture.

[0062] Step 5: The panoramic stitching module utilizes the image data collected by the multi-camera module and calibration data, etc., to perform panoramic stitching, obtaining a panoramic picture. This process ensures the integrity and consistency of the panoramic picture.

[0063] Step 6: The panoramic and close-up picture superimposed display module is responsible for the display mode of the panoramic picture and the close-up picture, as well as the switching processing of the close-up picture. Through this module, users can simultaneously view the panoramic picture and the close-up picture, obtaining a more rich and intuitive visual experience.

[0064] Through the above process, the device described in this embodiment can efficiently process audio and video data, achieving precise audio and video fusion and close-up picture processing, and providing users with high-quality audio-visual experience.

[0065] The device described in this embodiment realizes the acquisition of high-definition seamless close-up pictures, effectively solving the problem of the difficulty in balancing panoramic coverage and high-definition close-up, specifically: The device adopts a ring layout of multiple camera modules (not less than 2), combined with the logical design of "optional panoramic angle range + enlarged adjacent overlapping angle", which not only ensures the complete coverage of the panoramic picture, but also provides sufficient overlapping area for subsequent processing. This design greatly facilitates smooth image stitching and transition, improving the coherence of the picture.

[0066] Further, the device introduces the cooperative constraint condition of overlapping angle and boundary distance threshold. This strategy ensures that the boundary person is always near the center picture area of the adjacent camera module, effectively avoiding image distortion and stitching problems caused by the person approaching the boundary, and improving the overall quality and visual effect of the picture.

[0067] Further, in terms of panoramic field of view clipping, the device adopts a joint strategy of hardware and software. Through optimization at the hardware level and fine adjustment of software algorithms, efficient clipping of the panoramic field of view is achieved. This joint strategy not only improves the clipping efficiency, but also ensures the resolution and quality of the clipped picture, meeting the needs of different application scenarios.

[0068] In addition, based on the uniform circular microphone array, the device realizes rapid positioning and priority sorting of the speaker close-up lens. Through accurate sound source positioning and voice activity detection, the device can quickly identify the person speaking and prioritize according to their position and importance, ensuring the accuracy and timeliness of the close-up picture.

[0069] Embodiment Three: The embodiment provides a conference terminal device comprising the multi-camera module overlapping field of view configuration and close-up picture acquisition device of embodiment two.

[0070] Those skilled in the art will appreciate that embodiments of the application can be devised for a method, a system, or a computer program product. Accordingly, the present application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0071] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0072] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0073] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0074] Obviously, the above-described embodiments are only examples and are not intended to limit the present application. Based on the above description, those skilled in the art can make other variations and modifications of the present application without deviating from the scope of the present application. Therefore, the present application should not be limited by the above embodiments.

Claims

1. A multi-camera module overlapping field of view configuration and close-up picture acquisition method, characterized in that, The method comprises the following steps: S1. Arranging a plurality of camera modules in sequence along a preset direction and a preset field of view overlap area; obtaining a decoding stream output by each camera module, performing human body detection on the decoding stream respectively to obtain candidate human image coordinates corresponding to each decoding stream; and performing detection target deduplication processing on all candidate human image coordinates; S2. Converting the deduplicated candidate human image coordinates into corresponding image angles according to an audio-video angle mapping table to obtain an image angle set; S3. Obtaining an audio angle; comparing the audio angle with the image angle set according to a preset matching rule to obtain a matching combination; and determining a target object angle according to the matching combination; S4. Calculating a boundary distance of the target object angle in a field of view of each camera module to obtain a boundary distance set; S5. Screening out a maximum value in the boundary distance set, and obtaining a matching camera module according to the maximum value; extracting an output picture of the matching camera module, and obtaining a close-up picture of a target object according to the output picture.

2. The multi-camera module overlapping field of view configuration and close-up picture acquisition method of claim 1, wherein, In S3, the matching rule is: If no sound source signal is detected as human voice or no human body target is detected in the current frame, the current frame is directly skipped and no matching combination is determined; if there is human voice and human body detection result in the current frame, the image angle closest to the audio angle in the image angle set is selected as the matching combination.

3. The multi-camera module overlapping field of view configuration and close-up picture acquisition method of claim 1, wherein, In S4, the step of calculating the boundary distance of the target object angle in the field of view of each camera module is: obtaining a camera module field of view start angle; calculating a difference value between the target object angle and the camera module field of view start angle to obtain a first difference value; obtaining a camera module field of view end angle; calculating a difference value between the camera module field of view end angle and the target object angle to obtain a second difference value; comparing the first difference value and the second difference value, and selecting a minimum value as the boundary distance.

4. The multi-camera module overlapping field of view configuration and close-up picture acquisition method of claim 1, wherein, In S5, the process of extracting the output picture of the matching camera module and obtaining the close-up picture of the target object according to the output picture comprises: cropping a region where the target object is located from the output picture to obtain a cropped picture; performing distortion correction on the cropped picture by using overlap distortion correction, and performing smoothing processing on an edge region of the cropped picture by using a center weight map to obtain a close-up picture of the target object.

5. The multi-camera module overlapping field of view configuration and close-up picture acquisition method of claim 1, wherein, The method further comprises the following steps after S5: performing splicing processing on the close-up picture and non-close-up pictures; wherein the splicing processing step is: limiting a splicing range of a panoramic field of view, and performing splicing processing on camera module output pictures in the splicing range; setting a splicing seam of the close-up picture and a plurality of non-close-up pictures in a preset low attention area.

6. The multi-camera module overlapping field of view configuration and close-up picture acquisition method of claim 1, wherein, In S1, the preset field of view overlap area is 60°.

7. A multi-camera module overlapping field of view configuration and close-up picture acquisition apparatus for implementing the method of any one of claims 1 to 6, characterized in that, The method comprises the following steps: providing a plurality of camera modules to provide a plurality of original images, and outputting a corresponding decoding stream according to each original image; A human body detection module is configured to acquire a decoded stream output by each of the camera modules, perform human body detection on the decoded stream respectively to obtain candidate human image coordinates corresponding to each decoded stream, and perform detection target deduplication processing on all candidate human image coordinates. An image angle acquisition module is configured to convert the deduplicated candidate human image coordinates into corresponding image angles according to an audio-video angle mapping table to obtain an image angle set. An audio positioning module is configured to acquire an audio angle. According to a matching rule, the audio angle is compared with the image angle set to obtain a matching combination, and a target object angle is determined according to the matching combination. A calculation module is configured to calculate a boundary distance of the target object angle in a field of view of each camera module to obtain a boundary distance set. A close-up picture output module is configured to screen out a maximum value in the boundary distance set, obtain a matching camera module according to the maximum value, extract an output picture of the matching camera module, and obtain a close-up picture of a target object according to the output picture.

8. The multi-camera module overlapping field of view configuration and close-up picture acquisition apparatus according to claim 7, wherein, The plurality of camera modules are sequentially arranged along a preset annular direction.

9. The multi-camera module overlapping field of view configuration and close-up picture acquisition apparatus according to claim 8, wherein, The camera module array includes a first camera module, a second camera module, a third camera module, and a fourth camera module, a field of view coverage range of the first camera module is [30°, 150°), a field of view coverage range of the second camera module is [90°, 210°), a field of view coverage range of the third camera module is [150°, 270°), and a field of view coverage range of the fourth camera module is [210°, 330°).

10. The multi-camera module overlapping field of view configuration and close-up picture acquisition apparatus according to claim 7, wherein, The audio positioning module includes a microphone array, and the microphone array includes a circular ring and a plurality of microphones.

Images