A shooting method of a portable mobile shooting terminal with multiple CMOS combinations
By integrating multiple CMOS sensors and multiple optical lenses in portable mobile camera terminals, combining optical digital automatic zoom and digital zoom methods, shooting from macro to ultra-long distance is achieved, solving the poor flexibility of traditional practical training and recording and broadcasting vehicles. The problem of inconvenient lens replacement provides a portable, highly integrated film terminal and automatic zoom solution.
Patent Information
- Application Number
- CN202411173660.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-08-26
AI Technical Summary
Traditional training recording and broadcasting cars are large and have poor flexibility, and are difficult to achieve shooting from macro to ultra-long distance in teaching scenarios, and it is inconvenient to replace optical lenses, which is expensive.
Design a portable mobile camera terminal with multiple CMOS combinations to integrate multiple optical lenses with different magnifications, and automatically switch lenses and processing methods through optical number automatic zoom conversion and digital zoom methods to realize the lens and processing methods from macro to ultra-long distance. Shoot.
It realizes a portable high-integration filming terminal, which can perform seamless zooming at different magnifications, solves the problems of picture jump and zoom discontinuity, and meets the needs of multiple teaching scenarios.
Smart Images

Figure CN118695058B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of practical image shooting, and in particular relates to a portable mobile shooting terminal and a shooting method of a multi-CMOS combination. Background Art
[0002] With the widespread popularization of training recording and broadcasting terminals in various vocational and general education institutions, training recording and broadcasting vehicles of different types and sizes have gradually emerged in the market. At present, training recording and broadcasting vehicles generally adopt a larger body structure, and carry different types of cameras to shoot the target object and realize local storage, live streaming and other functions.
[0003] Traditional training recording vehicles have a large body, which is not convenient for flexible and mobile shooting. In addition, in actual teaching scenarios, teachers can shoot targets from macro to ultra-long distances according to the needs of the target, but it is not convenient to change the optical lens in real time during class. Existing training recording vehicles generally use multiple cameras with different focal lengths to achieve the function of shooting targets at different distances, which is costly. If only a single camera terminal is used, it is impossible to cover the shooting targets from macro to ultra-long distance. Summary of the invention
[0004] Purpose of the invention: In view of the problems existing in the above-mentioned background technology, the present invention provides a portable mobile filming terminal with a multi-CMOS combination, on the basis of which a plurality of optical lenses with different magnifications are integrated, and at the same time provides a filming method, which automatically switches the lens and the zoom mode when zooming in and out of the target through automatic zoom conversion of the optical number, thereby solving the problems of picture jumping and discontinuous zoom when actually zooming in and out of the target, and finally realizing that a set of system can complete the shooting of teaching scenes from macro to ultra-long distance.
[0005] Technical solution: In the first aspect, the present application provides a portable mobile camera terminal with a multi-CMOS combination, including several optical lenses and a main control chip, each optical lens is connected to a corresponding CMOS sensor; the main control chip integrates an ISP unit for image signal processing, a DSP module for digital signal processing, an NPU for AI enhancement processing, and a VPU for encoding and decoding; the ISP is connected to each CMOS sensor, selects the optical lens of the target channel and performs optical zoom, and collects the zoomed image; the DSP receives the ISP processing results and outputs the image frame after digital zoom and image enhancement as required.
[0006] Furthermore, the NPU receives the output result of the DSP and performs enhancement processing, transmits the processed image to the VPU for encoding and decoding, and completes the output of the video stream.
[0007] In a second aspect, the present invention provides a method for shooting a portable mobile shooting terminal with multiple CMOS combinations, comprising the following steps:
[0008] Step S1, identifying the user's magnification requirement. When the user's magnification is within the optical zoom range of a certain optical lens, the ISP selects the optical lens and sends a control instruction to perform optical zoom to obtain a target image.
[0009] Step S2: when the user magnification is between two adjacent optical lenses, the optimal optical lens is selected according to the user magnification, and the image to be processed is acquired at the closest magnification under the optimal optical lens, and the DSP zooms in or out the image to be processed to further acquire the target image;
[0010] Step S3, performing frame fusion and interpolation between the image to be processed obtained in step S2 and the target image, and outputting the fused interpolation frame image as a transition frame.
[0011] Furthermore, in step S2, the user magnification is set to a, and the magnification intervals of the n optical lenses are set to [x1, y1], [x2, y2], ..., [xn, yn], wherein y1 < x2, y2 < x3 ... yn-1 < xn; when a∈(yn-1, xn), further judgment is made: if a-(yn-1) ≥ (xn)-a, the nth optical lens is taken as the optimal optical lens, and the ISP selects to receive the output signal of the nth optical lens Lens n; otherwise, the n-1th optical lens is taken as the optimal optical lens, and the ISP selects to receive the output signal of the n-1th optical lens Lens n-1.
[0012] Furthermore, in step S2, after selecting the optimal optical lens, the target image information is obtained with a magnification closest to a, the ISP receives the corresponding optical image information and performs photoelectric conversion, and after obtaining the digital image information, transmits it to the DSP for digital zoom; when the optimal optical lens is Lens n, the DSP performs scaling processing on the target image captured by Lens n at a magnification of xn to obtain a target image with a magnification of a; when the optimal optical lens is Lens n-1, the DSP performs amplification and cropping processing on the target image captured by Lens n-1 at a magnification of yn-1 to obtain a target image with a magnification of a.
[0013] Furthermore, the image frame elements F0 (x, y) and F1 (x, y) before and after the digital zoom are fused and interpolated, and a supplementary frame pixel Ft (x, y) is generated between F0 (x, y) and F1 (x, y) by linear interpolation; the supplementary frame is calculated as follows:
[0014] Ft(x,y)=µ*F0(x,y)+(1-µ)*F1(x,y)
[0015] By changing the interpolation coefficient µ, multiple complementary frames Ft are generated i (x, y), where i is a positive integer;
[0016] Ft i (x, y) are fused into one frame. During the fusion process, all repeated pixels are filtered out, and the remaining pixels are retained and spliced to generate a transition frame. The DSP then performs color enhancement on the pixels as the final interpolated image.
[0017] Compared with the prior art solutions, the technical solution adopted by the present invention has the following beneficial effects:
[0018] (1) The present invention adopts a multi-CMOS combination method to integrate multiple optical lenses and a main control chip on the same portable mobile camera terminal, thereby ensuring the shooting performance while having the feature of portability.
[0019] (2) The present invention provides a shooting method combining optical zoom and digital zoom. Combined with a highly integrated multi-camera mobile terminal, it can cover the shooting scene requirements from wide angle to macro and realize automatic switching of optical and digital zoom methods under different magnifications.
[0020] (3) The present invention provides a method for automatically switching between optical zoom and digital zoom, which selects the optimal optical lens and performs automatic optical and digital zoom. At the same time, a DSP module is used to perform fusion interpolation to generate interpolated images for transition, making the zooming process smoother and solving the problem of picture jitter during the zooming process. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 A schematic diagram of the architecture of a portable mobile camera terminal system with multiple CMOS combinations provided by the present invention;
[0022] Figure 2 A data processing flow chart of the portable mobile camera terminal provided by the present invention;
[0023] Figure 3 This is a schematic diagram of the light-to-digital automatic zoom conversion method provided by the present invention. DETAILED DESCRIPTION
[0024] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, systems, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0025] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.
[0026] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0027] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.
[0028] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.
[0029] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.
[0030] It should be understood that the size of the serial numbers of the steps in this embodiment does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.
[0031] In order to solve the problem that traditional training recording and broadcasting equipment needs to prepare multiple camera terminals to adapt to teaching scenes at different distances during shooting, and at the same time solve the problem of selecting and applying for different magnification scenarios when multiple cameras are used at the same time, the present application provides a portable mobile shooting terminal with a multi-CMOS combination, and designs a portable shooting terminal with an integrated multi-CMOS combination. Through a highly integrated hardware design, one terminal can meet a variety of teaching scenes from wide angle to macro. In addition, a combined zoom and shooting method based on an automatic zoom conversion method of light numbers is provided, which automatically switches the optical lens when shooting targets at different distances, and assists the digital zoom method to achieve seamless switching of the picture. When performing digital zoom, frames are supplemented by a multi-frame interpolation method to achieve clearer and smoother zoom magnification. An embodiment is provided below in conjunction with the accompanying drawings. First, the working principle of the portable mobile shooting terminal system architecture provided by the present invention is explained.
[0032] like Figure 1-2 As shown, the present invention designs a portable mobile camera terminal with a multi-CMOS combination, and the mobile camera terminal includes several optical lenses and a main control chip. Each optical lens is connected to a corresponding CMOS sensor. The main control chip integrates an ISP unit for image signal processing, a DSP module for digital signal processing, an NPU for AI enhancement processing, and a VPU for encoding and decoding. The ISP receives the digital image signal output by the CMOS sensor through the MIPI or SLVS-ES interface and processes it. The ISP selects a suitable lens according to user needs, obtains the image information, and transmits it to the DSP module for digital signal processing. The DSP module automatically zooms the image according to demand, optimizes the zoomed image frame, and transmits it to the NPU after processing for AI processing such as intelligent scene recognition, and finally completes the output of the video stream after encoding and decoding by the VPU.
[0033] The present invention uses multiple CMOS sensors corresponding to multiple optical lenses to collect images, and a highly integrated main control chip is responsible for collecting and processing images transmitted by each camera. The main control chip not only includes traditional CPU and GPU hardware, but also integrates multiple processing cores including ISP, DSP, NPU and VPU, which ensures the comprehensiveness of functions while realizing the portability of the mobile shooting terminal.
[0034] like Figure 3As shown, this embodiment provides an automatic zoom method in the integrated 3-camera state, through optical magnification and automatic hybrid zoom of different optical lenses, real-time switching of near and far is achieved according to different distances of the target object, and continuous and smooth zooming of the picture is achieved. Specifically,
[0035] In this embodiment, the first optical lens Lens1 is a 1x optical fixed focus lens, the second optical lens Lens2 is an optical zoom lens with a magnification of 5-10 times, and the third optical lens Lens3 is an optical zoom lens with a magnification of 15-20 times. With Lens1 as the main camera, when the user takes a photo, the ISP receives the CMOS sensor image data corresponding to Lens1 and processes the image information, then outputs the digital image signal to the DSP, which performs digital signal processing as needed. Then it is transmitted to the NPU for AI enhancement processing, and then the VPU performs encoding and decoding processing and outputs the video stream.
[0036] When the user is shooting macro and wide-angle, Lens1 can meet the scene. When the user zooms in, the ISP receives image data from different channels according to different magnification requirements. When the magnification is 5-10 times, the ISP chooses to receive Lens2, adjusts to the corresponding magnification through optical zoom, and transmits the corresponding CMOS sensor image data to the ISP, and outputs the video stream after processing using the same method. When the magnification is 15-20 times, the ISP chooses to receive Lens3, and also uses the above method to process and output the corresponding video stream.
[0037] During actual shooting, since the magnification of the three optical lenses cannot cover the entire magnification range, when the user's magnification is 1-5 times or 10-15 times, only using the traditional optical zoom method will not only cause the problem of blurred imaging, but also the problem of picture jumping and unsmooth transition during actual continuous zooming. The present application provides an automatic zoom method that combines digital zoom with optical zoom. When the user's zoom in / out ratio is between the optical zoom ratios of several cameras, the most appropriate optical lens is selected according to the actual ratio. After receiving the image information of the optical lens, the digital zoom method is used to further zoom in / out the image, and the corresponding target image is output, ultimately achieving a continuous and smooth zoom effect. Specifically,
[0038] Step S1, obtaining the current magnification a, selecting adjacent magnification intervals according to the magnification, and selecting the optimal optical lens.
[0039] The magnification intervals of the n optical lenses are set to [x1, y1], [x2, y2], ..., [xn, yn], where y1 < x2, y2 < x3 ... yn-1 < xn.
[0040] When a∈(yn-1,xn), further judgment is made: if a-(yn-1)≥(xn)-a, the nth optical lens is the optimal optical lens, and the ISP chooses to receive the output signal of the nth optical lens Lens n; otherwise, the n-1th optical lens is the optimal optical lens, and the ISP chooses to receive the output signal of the n-1th optical lens Lens n-1.
[0041] Step S2: acquiring image information based on the optimal optical lens, and performing digital zoom according to the image information to acquire a target image at a magnification a.
[0042] Step S2.1, after confirming the optimal optical lens, the target image information is obtained with the magnification closest to a. The ISP receives the corresponding optical image information and performs photoelectric conversion. After obtaining the digital image information, it is transmitted to the DSP for digital zoom. When the optimal optical lens is Lens n, the DSP performs zoom processing on the target image captured by Lens n at a magnification of xn to obtain the target image with a magnification of a. When the optimal optical lens is Lens n-1, the DSP performs zoom and crop processing on the target image captured by Lens n-1 at a magnification of yn-1 to obtain the target image with a magnification of a.
[0043] Step S2.2: perform fusion interpolation on the image frame elements F0 (x, y) and F1 (x, y) before and after digital zoom to solve the problem of image jitter and reduced smoothness when the image is transformed before and after. Specifically,
[0044] The supplementary frame pixel Ft(x, y) is generated by linear interpolation between F0(x, y) and F1(x, y). The supplementary frame is calculated as follows:
[0045] Ft(x,y)=µ*F0(x,y)+(1-µ)*F1(x,y)
[0046] By changing the interpolation coefficient µ, multiple complementary frames Ft are generated i (x, y), where i is a positive integer.
[0047] Ft i (x, y) are fused into one frame. When fusion is performed, all duplicate pixels are filtered out, and the remaining pixels are retained and stitched. The DSP performs color enhancement on the pixels as the final interpolated image.
[0048] The interpolated frame image Ft is inserted between the image frames F0 and F1 before and after the digital zoom as a transition frame to achieve a smooth transition between the optical zoom and the digital zoom and solve the problem of image frame jumps before and after the transformation.
[0049] Through the portable mobile camera terminal with a multi-CMOS combination designed in the present application, it is possible to cover a variety of application scenarios from wide-angle macro to long focal length under multi-camera conditions. Under different magnification factors, the terminal can automatically select the optimal optical lens for image reception, processing, and display, truly realizing the seamless switching of cameras during zooming in and out, and solving the problems of picture jumping and discontinuous zoom that may exist in the transition focal length of existing multi-camera solutions.
[0050] It should be understood that the size of the sequence number of each step in the above-described embodiment does not mean the order of execution, and the order of execution of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiment of the present application. In the description, for the purpose of explanation rather than for limitation, specific details such as specific system structure, technology and the like are proposed, so as to thoroughly understand the embodiment of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, the detailed description of well-known systems, devices, circuits and methods is omitted to prevent unnecessary details from hindering the description of the present application.
[0051] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.
[0052] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0053] Additionally, in the specification of the present application and the description of the appended claims, the terms “include,” “comprise,” “have,” and their variations all mean “including but not limited to,” unless otherwise specifically emphasized.
[0054] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0055] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0056] In the embodiments provided in the present application, it should be understood that the disclosed devices, computer equipment and methods can be implemented in other ways. For example, the device and computer equipment embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0057] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A shooting method of a portable mobile shooting terminal with multiple CMOS combinations, characterized in that: The portable mobile camera terminal with multiple CMOS combinations includes several optical lenses and a main control chip, each of which is connected to a corresponding CMOS sensor; the main control chip integrates an ISP unit for image signal processing, a DSP module for digital signal processing, an NPU for AI enhancement processing, and a VPU for encoding and decoding; the ISP is connected to each CMOS sensor, selects the optical lens of the target channel and performs optical zoom, and collects the zoomed image; The DSP receives the ISP processing result and performs digital zoom and image enhancement as required, then outputs the image frame; the NPU receives the DSP output result and performs enhancement processing, transmits the processed image to the VPU for encoding and decoding, and completes the output of the video stream; The shooting method of the portable mobile shooting terminal with multiple CMOS combinations comprises the following steps: Step S1, identifying the user's magnification requirement. When the user's magnification is within the optical zoom range of a certain optical lens, the ISP selects the optical lens and sends a control instruction to perform optical zoom to obtain a target image. Step S2: when the user magnification is between two adjacent optical lenses, the optimal optical lens is selected according to the user magnification, and the image to be processed is acquired at the closest magnification under the optimal optical lens, and the DSP zooms in or out the image to be processed to further acquire the target image; Specifically, the user magnification is set to a, and the magnification intervals of the n optical lenses are set to [x1, y1], [x2, y2], ..., [xn, yn], where y1 < x2, y2 < x3 ... yn-1 < xn; when a∈(yn-1, xn), further judgment is made: if a-(yn-1) ≥ (xn)-a, the nth optical lens is taken as the optimal optical lens, and the ISP chooses to receive the output signal of the nth optical lens Lensn; otherwise, the n-1th optical lens is taken as the optimal optical lens, and the ISP chooses to receive the output signal of the n-1th optical lens Lens n-1; Step S3, performing frame fusion and interpolation between the image to be processed obtained in step S2 and the target image, and outputting the fused interpolation frame image as a transition frame.
2. The shooting method of the portable mobile shooting terminal with multiple CMOS combinations according to claim 1, characterized in that: In the step S2, after selecting the optimal optical lens, the target image information is obtained with a magnification closest to a, the ISP receives the corresponding optical image information and performs photoelectric conversion, and after obtaining the digital image information, transmits it to the DSP for digital zoom; when the optimal optical lens is Lens n, the DSP performs scaling processing on the target image captured by Lens n at a magnification of xn to obtain a target image with a magnification of a; when the optimal optical lens is Lens n-1, the DSP performs amplification and cropping processing on the target image captured by Lens n-1 at a magnification of yn-1 to obtain a target image with a magnification of a.
3. The shooting method of the portable mobile shooting terminal with multiple CMOS combinations according to claim 1, characterized in that: The image frame elements F0 (x, y) and F1 (x, y) before and after digital zoom are fused and interpolated, and a supplementary frame pixel Ft (x, y) is generated between F0 (x, y) and F1 (x, y) by linear interpolation; the supplementary frame is calculated as follows: Ft(x,y)=µ*F0(x,y)+(1-µ)*F1(x,y) By changing the interpolation coefficient µ, multiple complementary frames Ft are generated i (x, y), where i is a positive integer; Ft i (x, y) are fused into one frame. During the fusion process, all repeated pixels are filtered out, and the remaining pixels are retained and spliced to generate a transition frame. The DSP then performs color enhancement on the pixels as the final interpolated image.
