Control method of multiple camera components, computer equipment and storage medium

Abstract

The invention discloses a control method of multiple camera components, computer equipment and a storage medium. The control method of the multiple camera components comprises the following steps: receiving a first request instruction of calling the multiple camera components by a first application; and in response to the first request instruction, establishing and executing a first virtual pipeline, the first virtual pipeline comprising a plurality of first virtual pipeline processing stages, the plurality of first virtual pipeline processing stages respectively calling one of the plurality of camera components to execute the first request instruction. According to the control method for the multiple camera components, the multiple components of the same camera can be called by the multiple applications, and when the multiple applications call the multiple camera components at the same time, the problem of picture interruption or obvious frame dropping does not occur.

Description

Technical Field

[0001] The present application relates to the field of computer vision measurement technology, and in particular to a control method for multiple camera components, a computer device, and a storage medium. Background Art

[0002] In related technologies, cameras, in addition to being used in consumer-grade mobile phones and tablets, are also becoming widely used in security, video conferencing, automotive, and other product fields, and they play a core role in these products. Due to the increasing complexity of camera applications in existing technologies, they also face difficult problems. For example, it is difficult for multiple applications to use the same camera simultaneously. If another application calls the camera during operation, the current camera workflow will be interrupted or significant frame drops will occur. Summary of the Invention

[0003] In view of this, the present application provides a control method, computer equipment and storage medium for multiple camera components to solve the problem in the prior art that the same camera is difficult to be used by multiple applications at the same time. If other applications call the camera during the use of the camera, the current camera workflow will be interrupted or obvious frame drops will occur.

[0004] In order to solve the above technical problems, the first technical solution provided in this application is: providing a control method for multiple camera components, comprising the following steps: receiving a first request instruction from a first application to call the multiple camera components; and establishing and executing a first virtual pipeline in response to the first request instruction, wherein the first virtual pipeline includes multiple first virtual pipeline processing stages, and the multiple first virtual pipeline processing stages respectively call one of the multiple camera components to execute the first request instruction.

[0005] Preferably, the control method of multiple camera assemblies further includes the following steps:

[0006] Receiving a second request instruction of at least one second application to call the plurality of camera components; wherein the second application is different from the first application;

[0007] In response to the second request instruction, a second virtual pipeline is established and executed, wherein the second virtual pipeline includes multiple second virtual pipeline processing stages, and the multiple second virtual pipeline processing stages respectively call one of the multiple camera components to execute the second request instruction; wherein, at least one of the multiple second virtual pipeline processing stages is the same as the camera component called by at least one of the multiple first virtual pipeline processing stages.

[0008] Preferably, the control method of multiple camera assemblies further includes the following steps:

[0009] Receiving a second request instruction of at least one second application to call the plurality of camera components; wherein the second application is different from the first application;

[0010] In response to the second request instruction, a second virtual pipeline is established and executed, wherein the second virtual pipeline includes multiple second virtual pipeline processing stages, and the multiple second virtual pipeline processing stages respectively call one of the multiple camera components to execute the second request instruction, wherein the multiple second virtual pipeline processing stages are the same as the multiple first virtual pipeline processing stages.

[0011] Preferably, the control method of multiple camera assemblies further includes the following steps:

[0012] Creating at least one first physical pipeline corresponding to the multiple camera components; wherein the first physical pipeline includes multiple first physical pipeline processing stages, the multiple first virtual pipeline processing stages correspond to the multiple first physical pipeline processing stages, and the multiple first physical pipeline processing stages respectively correspond to the multiple camera components;

[0013] Create at least one second physical pipeline corresponding to the multiple camera components; wherein the second physical pipeline includes multiple second physical pipeline processing stages, the multiple second virtual pipeline processing stages correspond to the multiple second physical pipeline processing stages, and the multiple second physical pipeline processing stages respectively correspond to the multiple camera components.

[0014] Preferably, while executing the second virtual pipeline, unfinished related operations of the first virtual pipeline are executed.

[0015] Preferably, the step of establishing and executing the second virtual pipeline in response to the second request instruction includes:

[0016] Switching at least one of the plurality of camera components to execute the second request instruction;

[0017] The step of executing the unfinished related operations of the first virtual pipeline while executing the second virtual pipeline includes:

[0018] The remaining components of the multiple camera components corresponding to the multiple first virtual pipeline processing stages continue to perform the unfinished related operations of the first virtual pipeline.

[0019] Preferably, the first request instruction and the second request instruction come from different multiple applications respectively; wherein the first virtual pipeline and the second virtual pipeline of the multiple applications run independently of each other.

[0020] Preferably, before establishing the first virtual pipeline and the second virtual pipeline corresponding to the multiple applications, the method includes:

[0021] determining whether a plurality of identical camera components corresponding to a plurality of physical pipeline processing stages in the first physical pipeline and the second physical pipeline have support parameters corresponding to each of the applications;

[0022] In response to the plurality of camera components having support parameters corresponding to each of the applications, the support parameters of the plurality of camera components corresponding to the first virtual pipeline and the second virtual pipeline are the same;

[0023] In response to the plurality of camera components not having support parameters corresponding to each of the applications, the support parameters of the plurality of camera components corresponding to the first virtual pipeline and the second virtual pipeline are different.

[0024] Preferably, the control method of the plurality of camera assemblies further includes:

[0025] The last frame image of the first virtual pipeline and the first frame image of the second virtual pipeline are output in sequence.

[0026] Preferably, the plurality of camera components corresponding to the plurality of first physical pipeline processing stages and the plurality of second physical pipeline processing stages include one or more of image source acquisition, image noise reduction, picture capture and distortion-aware quantization.

[0027] In order to solve the above technical problems, the second technical solution provided in this application is: to provide a computer device, including: a processor and a memory, the memory is connected to the processor, and is used to store a computer program that can be run on the processor; wherein, when the processor executes the computer program, it implements any of the methods described above.

[0028] In order to solve the above technical problems, the third technical solution provided in this application is: providing a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements any of the methods described above.

[0029] Beneficial effects of the present application: Different from the prior art, the control method of multiple camera components of the present application includes: receiving a first request instruction from a first application to call multiple camera components; and in response to the first request instruction, establishing and executing a first virtual pipeline, wherein the first virtual pipeline includes multiple first virtual pipeline processing stages, and the multiple first virtual pipeline processing stages respectively call one of the multiple camera components to execute the first request instruction. The control method of multiple camera components provided by the present application enables multiple components of the same camera to be called by multiple applications, and when multiple applications call multiple camera components simultaneously, there will be no image interruption or obvious frame drop problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0031] Figure 1 This is a flowchart of the overall control method for multiple camera components provided by this application;

[0032] Figure 2 A flowchart of a method for controlling multiple camera assemblies according to the first embodiment of the present application;

[0033] Figure 3 A flowchart of a method for controlling multiple camera assemblies according to a second embodiment of the present application;

[0034] Figure 4 A schematic diagram of the interactive flow of the control method for multiple camera components provided in this application;

[0035] Figure 5 A flowchart of a comparative example of multiple virtual pipelines working simultaneously provided by this application;

[0036] Figure 6 A flowchart of the workflow of multiple virtual pipelines working simultaneously provided by this application;

[0037] Figure 7 A flowchart of a method for controlling multiple camera assemblies according to a third embodiment of the present application;

[0038] Figure 8 A flowchart of the steps before establishing the first virtual pipeline and the second virtual pipeline corresponding to multiple applications provided by the present application;

[0039] Figure 9A flowchart of steps for creating at least one second physical pipeline corresponding to multiple camera components provided in any embodiment of the present application;

[0040] Figure 10 A simplified structural diagram of a hardware integrator with multiple camera components provided in this application;

[0041] Figure 11 This is a schematic diagram of the operation of calling multiple camera components when two virtual pipelines work simultaneously in an example provided by this application;

[0042] Figure 12 A simplified diagram of the structure of the support parameters for multiple applications to call the physical pipeline simultaneously provided by this application;

[0043] Figure 13 is a structural diagram of a computer device provided in one embodiment of the present application;

[0044] Figure 14 This is a schematic block diagram of the structure of a computer-readable storage medium provided in one embodiment of the present application. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0046] The terms "first", "second" and "first" in this application are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" can explicitly or implicitly include at least one of these features. In the embodiments of the present application, all directional indications (such as up, down, left, right, front, back ...) are only used to explain the relative positional relationship, movement, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication also changes accordingly. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the steps or units listed, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0047] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0048] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the individual of the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, a clear and prominent sign is set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that they agree to the collection of their personal information; or on the personal information processing device, when the personal information processing rules are notified by obvious signs / information, the individual's authorization is obtained through pop-up information or by asking the individual to upload their personal information; among which, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0049] The inventors of this application have discovered that, in the prior art, it is difficult for multiple applications to use the same camera simultaneously. If another application calls the camera while it is in use, the current camera workflow will be interrupted or there will be obvious frame drops. At the same time, after changing the camera pipeline, the pipeline workflow needs to be reconfigured, and the original camera pipeline will be completely interrupted. After reopening the camera pipeline, it will be found that the camera pipeline screen is disconnected for several frames. In particular, when applied to in-vehicle virtualized camera operations, there may be problems with the operating system not supporting it, making it difficult to virtualize the camera.

[0050] In order to solve the above problems, the present application provides a control method for multiple camera components.

[0051] See also Figure 1 and Figure 10 , Figure 1 This is a flowchart of the overall control method for multiple camera components provided by this application; Figure 10 A simplified structural diagram of a hardware integrator with multiple camera components provided in this application.

[0052] The control method of multiple camera components provided in this application may include the following steps:

[0053] S1: Receive a first request instruction from a first application to call multiple camera components.

[0054] Specifically, the multiple camera components 10 are connected to the controller 30 and receive application call instructions through the controller 30. When the first application calls the multiple camera components 10, the controller 30 responds. The multiple camera components 10 can be understood as multiple components for camera setup in the hardware integrator 100. The multiple camera components 10 can be called together or separately.

[0055] S2: In response to the first request instruction, establish and execute a first virtual pipeline, wherein the first virtual pipeline includes multiple first virtual pipeline processing stages 41, and the multiple first virtual pipeline processing stages 41 respectively call one of the multiple camera components 10 to execute the first request instruction.

[0056] Specifically, after the controller 30 responds to the first request instruction, it establishes a first virtual pipeline to process the first request instruction. Dividing the first virtual pipeline into multiple processing stages facilitates processing of the first virtual pipeline at different processing stages. For example, only a portion of the first virtual pipeline stages may be used to call one or more of the multiple first camera components at a time, rather than calling the camera component 10 through all first virtual pipeline processing stages 41. This allows other unused first virtual pipeline processing stages 41 to execute other camera components 10.

[0057] See also Figure 2 , Figure 2 This is a flowchart of a method for controlling multiple camera components provided in the first embodiment of the present application.

[0058] In the first embodiment provided in this application, the following steps may be further included based on the above steps:

[0059] S3: Create at least one first physical pipeline corresponding to multiple camera components 10.

[0060] Specifically, the first physical pipeline includes multiple first physical pipeline processing stages. Multiple first virtual pipeline processing stages 41 correspond to multiple first physical pipeline processing stages. One physical pipeline processing stage can correspond to multiple virtual pipeline processing stages. The multiple first physical pipeline processing stages correspond to multiple camera assemblies 10, for example, camera assemblies 10 may include 10-1, 10-2, 10-3, ..., 10-n. A camera assembly 10 can only be connected to one first physical pipeline processing stage at a time.

[0061] See also Figures 3 to 6 as well as Figures 11 and 12 , Figure 3A flowchart of a method for controlling multiple camera assemblies according to a second embodiment of the present application; Figure 4 A schematic diagram of the interactive flow of the control method for multiple camera components provided in this application; Figure 5 A flowchart of a comparative example of multiple virtual pipelines working simultaneously provided by this application; Figure 6 This is a flowchart of the workflow of multiple virtual pipelines working simultaneously provided by this application; wherein, Figure 5 and Figure 6 The shaded areas in the figure are inactive virtual pipeline processing stages; Figure 11 This is a schematic diagram of the operation of calling multiple camera components when two virtual pipelines work simultaneously in an example provided by this application; Figure 12 A simplified structural diagram of the support parameters for multiple applications to simultaneously call the physical pipeline provided by this application.

[0062] The control method of multiple camera assemblies 10 provided in the second embodiment of the present application may include the following steps:

[0063] S4A: Receive a second request instruction for at least one second application to call multiple camera components 10; wherein the second application is different from the first application.

[0064] Specifically, the second application is different from the first application, and multiple camera components 10 are called by the second application, that is, by an application other than the first application. In this case, the controller 30 needs to process the second request instruction of the second application on the basis of processing the first application calling multiple camera components 10. It is understood that the second application is a general term for other applications different from the first application, and the second application can be one or more.

[0065] S5A: In response to the second request instruction, establish and execute a second virtual pipeline, wherein the second virtual pipeline includes multiple second virtual pipeline processing stages 42, and the multiple second virtual pipeline processing stages 42 respectively call one of the multiple camera components 10 to execute the second request instruction; wherein, at least one of the multiple second virtual pipeline processing stages 42 is the same as the camera component 10 called by at least one of the multiple first virtual pipeline processing stages 41.

[0066] Specifically, such as Figure 4 、 Figure 6 as well as Figure 10 and Figure 11As shown, the establishment and execution process of the second virtual pipeline is basically the same as that of the first virtual pipeline, and the second virtual pipeline also includes multiple second virtual pipeline processing stages 42, and calls multiple camera components 10 through some or all of the multiple virtual pipeline processing stages. One or more of them can be called to execute the second request instruction of the second application. The camera component 10 called by the second virtual pipeline processing stage 42 has overlapping parts with the camera component 10 called by the first virtual pipeline processing stage 41, so it is necessary to perform runtime management for the camera components 10 that are called multiple times or repeatedly. When calling multiple camera components 10 in the second virtual pipeline processing stage 42, the camera components 10 called by the first virtual pipeline processing stage 41 can be the same or different. The camera component 10 called by the second virtual pipeline processing stage 42 is the same as the camera component 10 called by the first virtual pipeline processing stage 41 in at least one part, and the others can be the same or different. This application does not impose any restrictions on this.

[0067] For example, Figure 11 As shown, when the camera assembly 10 is in dual-camera mode, the first virtual pipeline is in operation. At this time, the first virtual pipeline calls the camera assembly 10 to work. After the first stage of the first virtual pipeline calls the camera assembly 10 (for example, the P1 stage) to complete, the camera assembly 10 of the first stage is in an inactive or idle state. Then, the second virtual pipeline in single-camera mode can call the camera assembly 10 of the first stage (for example, the P1 stage) to work. The second stage of the first virtual pipeline (for example, the WPE stage) continues to work, and the second virtual pipeline works immediately following the first virtual pipeline. After the second stage of the first virtual pipeline completes the call of the camera component 10 (for example, the WPE stage), the second virtual pipeline can then call the camera component 10 of the second stage. After all the working stages of the first virtual pipeline are completed, the second virtual pipeline can also complete the call of the camera component 10 in the shortest time, thereby saving the waiting and connection time between the two virtual pipelines to the greatest extent, and outputting the video frames processed by the first virtual pipeline and the second virtual pipeline in sequence to achieve the greatest degree of seamless connection.

[0068] In this embodiment, the following steps may be further included:

[0069] S6A: Create at least one second physical pipeline corresponding to the multiple camera components 10.

[0070] Specifically, the second physical pipeline includes multiple second physical pipeline processing stages. The multiple second virtual pipeline processing stages 42 correspond to the multiple second physical pipeline processing stages. One physical pipeline processing stage can correspond to multiple virtual pipeline processing stages. The multiple second physical pipeline processing stages correspond to multiple camera assemblies 10, respectively. A camera assembly 10 can only be connected to one second physical pipeline processing stage at a time.

[0071] The second physical pipeline and the first physical pipeline can have the same properties, and multiple second physical pipelines can generally refer to other physical pipelines other than the first physical pipeline. For example, when multiple second applications call multiple physical pipelines simultaneously, since the first physical pipeline is called and occupied by the first virtual pipeline processing stage 41, the call instructions of the multiple second applications can be responded to one by one through the multiple second physical pipelines. It is understood that the first physical pipeline and the second physical pipeline here can respectively correspond to multiple identical or different physical pipeline processing stages.

[0072] like Figure 5 As shown, the camera component 10 call process in the prior art must be completed after the previous virtual pipeline is processed or the interrupt process of the virtual pipeline is executed, and the physical pipeline and the virtual pipeline in the prior art are in a one-to-one correspondence. Each physical pipeline can only be called by one virtual pipeline at a time. Therefore, the virtual pipelines in the prior art cannot run in parallel. If the current virtual pipeline is not completed, the next virtual pipeline cannot be established, which will either cause the call to fail or cause the previous virtual pipeline to be interrupted. In response to the above problems, as Figure 6 As shown, the present application sets up multiple virtual pipelines, and each virtual pipeline can correspond to a physical pipeline, so that multiple camera components 10 can respond to call requests from multiple applications, and multiple virtual pipelines can execute multiple request instructions in parallel, thereby saving time and improving work efficiency. Figure 12In the illustrated embodiment, the physical pipeline includes multiple physical pipeline stages, namely, image source acquisition, image noise reduction, frame capture, warping, and floppy drive processing. In conventional technology, this physical pipeline can only support a single virtual pipeline, such as the first virtual pipeline of a first application, the second virtual pipeline of a second application, or the third virtual pipeline of a third application. In the embodiments of the present application, different physical pipeline stages of the same physical pipeline can be simultaneously called by multiple virtual pipelines. For example, the frame capture stage can be called by the third virtual pipeline of a third application, the warping stage can be called by the first virtual pipeline of the first application, and the floppy drive processing can be called by the second virtual pipeline of the second application. This solves the technical problem in the prior art where the physical components of a camera cannot respond to calls from multiple applications, improves the processing efficiency of the camera system, and avoids the technical problem of delays and frame drops caused by a single physical pipeline only supporting a single virtual pipeline, which results in the virtual pipeline needing to perform shutdown and reconfiguration processes when multiple applications call camera components.

[0073] See also Figure 7 , Figure 7 This is a flowchart of a method for controlling multiple camera components provided in the third embodiment of the present application.

[0074] The control method of multiple camera assemblies 10 provided in the third embodiment of the present application may include the following steps:

[0075] S4B: Receive a second request instruction for at least one second application to call multiple camera components 10; wherein the second application is different from the first application.

[0076] S5B: In response to the second request instruction, establish and execute a second virtual pipeline, wherein the second virtual pipeline includes multiple second virtual pipeline processing stages 42, and the multiple second virtual pipeline processing stages 42 respectively call one of the multiple camera components 10 to execute the second request instruction, wherein the multiple second virtual pipeline processing stages 42 are the same as the multiple first virtual pipeline processing stages 41.

[0077] Specifically, this embodiment is substantially the same as the second embodiment, differing in that the multiple second virtual pipeline processing stages 42 in this embodiment are identical to the multiple first virtual pipeline processing stages 41. That is, in this embodiment, when the second virtual pipeline processing stage 42 calls multiple camera assemblies 10, it calls the exact same camera assemblies 10 as the first virtual pipeline processing stage 41. For example, if there are ten camera assemblies 10 in total, the first virtual pipeline processing stage 41 calls the first through fifth camera assemblies 10, and the second virtual pipeline processing stage 42 also calls the first through fifth camera assemblies 10-1 through 10-5. In this case, the order in which the first through fifth camera assemblies 10-1 through 10-5 are used must be managed. For example, after the first virtual pipeline processing stage 41 calls the first camera assembly 10-1, the second virtual pipeline processing stage 42 then calls the first camera assembly 10-1, and so on. The remaining camera assemblies 10 can also be called in the same order.

[0078] S6B: Create at least one second physical pipeline corresponding to the multiple camera components 10.

[0079] Specifically, the second physical pipeline includes multiple second physical pipeline processing stages. The multiple second virtual pipeline processing stages 42 correspond to the multiple second physical pipeline processing stages. One physical pipeline processing stage can correspond to multiple virtual pipeline processing stages. The multiple second physical pipeline processing stages correspond to multiple camera assemblies 10, respectively. A camera assembly 10 can only be connected to one second physical pipeline processing stage at a time.

[0080] The multiple camera components 10 corresponding to the multiple first physical pipeline processing stages and the multiple second physical pipeline processing stages may include one or more of Image Source (obtaining image source), MSNR (maximum-signal-noise ratio, image noise reduction), Dptz Capture (screen capture) and Warp PQ (warped perceptual quantization).

[0081] like Figure 4 As shown, the first request instruction and the second request instruction come from different applications respectively.

[0082] In other words, multiple different applications call multiple camera assemblies 10, and the controller 30 responds to the request instructions of the different multiple applications to establish and execute multiple virtual pipelines, including a first virtual pipeline and a second virtual pipeline. The first and second virtual pipelines of the multiple applications run independently and do not interfere with each other. It is understood that the execution of the second request instruction does not affect the execution of the first request instruction, and the two can run in parallel.

[0083] It can be seen from the above embodiments that in any of the above embodiments provided in this application, if Figure 6 As shown, the first virtual pipeline includes multiple first virtual pipeline processing stages 41, the first physical pipeline includes multiple first physical pipeline processing stages, the multiple first virtual pipeline processing stages 41 correspond to multiple first physical pipeline processing stages, and the multiple first physical pipeline processing stages correspond to multiple camera components 10 respectively.

[0084] The second virtual pipeline includes multiple second virtual pipeline processing stages 42, the second physical pipeline processing stage includes multiple second physical pipeline processing stages, the multiple second virtual pipeline processing stages 42 correspond to multiple second physical pipeline processing stages, and the multiple second physical pipeline processing stages correspond to multiple camera components 10 respectively.

[0085] See also Figure 8 , Figure 8 This is a flowchart of the steps provided in this application before establishing the first virtual pipeline and the second virtual pipeline corresponding to multiple applications.

[0086] In any embodiment, before establishing the first virtual pipeline and the second virtual pipeline corresponding to the multiple applications, the following steps may be included:

[0087] S01: Determine whether a plurality of identical camera components 10 corresponding to a plurality of physical pipeline processing stages in a first physical pipeline and a second physical pipeline have support parameters corresponding to each application.

[0088] Specifically, it can be understood that the camera component 10 corresponding to the physical pipeline processing stage must have corresponding support parameters with the corresponding application in order for the application to call the camera component 10; otherwise, the call may fail. Simply put, the physical pipeline is a physical hardware product. The physical pipeline processing stage has multiple support parameters, and a physical pipeline can support multiple virtual pipelines operating simultaneously. Each operation of the virtual pipeline processing stage calling the camera component 10 relies on the underlying support of the physical pipeline to be implemented. Therefore, the camera component 10 must have corresponding support parameters with each application that calls it, so that the application can successfully call the corresponding camera component 10. At the same time, the application must also have corresponding support parameters with the physical pipeline so that the physical pipeline can support the corresponding application to execute the corresponding operation instructions. For example, the physical pipeline support parameters include Image Process, MSNR (maximum-signal-noise ratio, image noise reduction), Dptz Capture, and WarpPQ (warp-perceptual quantization). Therefore, the application that calls these multiple camera components 10 must also have at least one of these support parameters. It is understood that the controller 30 can be used to detect whether the application and the camera assembly 10 it calls have corresponding support parameters with the physical pipeline. Figure 12 As shown, the first application, the second application and the third application need to have the same support parameters as the physical pipeline in order to call the camera component 10 corresponding to the physical pipeline. Figure 12 In the example, the first, second, and third applications can simultaneously call different parameters of the physical pipeline without having to wait for other applications to complete their work before calling, thereby improving the efficiency of the camera component 10 and saving waiting time. When multiple applications need to call the camera component with the same parameters, they need to wait until the previous application has completed its call before calling.

[0089] S02: In response to the plurality of camera components 10 having support parameters corresponding to each application, the support parameters of the plurality of camera components 10 corresponding to the first virtual pipeline and the second virtual pipeline are the same.

[0090] Specifically, the controller 30 detects that the camera component 10 has corresponding support parameters for each application, and the corresponding multiple camera components 10 have the same support parameters through the first virtual pipeline formed by responding to the first request instruction and the second virtual pipeline formed by responding to the second request instruction.

[0091] S03: In response to the plurality of camera components 10 not having support parameters corresponding to each application, the support parameters of the plurality of camera components 10 corresponding to the first virtual pipeline and the second virtual pipeline are different.

[0092] Specifically, if the support parameters of the plurality of camera components 10 are different from the support parameters corresponding to each application, the support parameters of the camera components 10 forming the first virtual pipeline and the camera components 10 forming the second virtual pipeline are also different.

[0093] In the second embodiment and the third embodiment provided in the present application, in response to the second request instruction, step S5A or S5B of establishing and executing the second virtual pipeline may include:

[0094] At least one of the plurality of camera assemblies 10 is switched to execute the second request instruction.

[0095] Specifically, as mentioned above, the first virtual pipeline can include multiple first virtual pipeline processing stages 41. In the process of multiple first virtual pipeline processing stages 41 executing the first request instruction in sequence, the camera component 10 called by the first virtual pipeline processing stage 41 that has completed the instruction will be in an idle state. At this time, the multiple second virtual pipeline processing stages 42 of the second virtual pipeline can call the idle camera component 10 in sequence to execute the operations of the corresponding processing stages and complete the second request instruction.

[0096] See also Figure 9 , Figure 9 A flowchart of the steps of creating at least one second physical pipeline corresponding to multiple camera components provided in any embodiment of the present application.

[0097] In any of the first to third embodiments provided in this application, the following steps may also be included:

[0098] S7: While executing the second virtual pipeline, execute the unfinished related operations of the first virtual pipeline.

[0099] That is, when the second virtual pipeline is executed, the operation of the first virtual pipeline will not be interrupted.

[0100] In a further embodiment, step S7 of executing the unfinished related operations of the first virtual pipeline while executing the second virtual pipeline may include:

[0101] S71: The remaining components of the multiple camera components 10 corresponding to the multiple first virtual pipeline processing stages 41 continue to perform related operations of the unfinished first virtual pipeline.

[0102] Specifically, as described above, the multiple second virtual pipeline processing stages 42 of the second virtual pipeline can sequentially call multiple camera components 10 in an idle state to execute the second request instruction. For example, the first virtual pipeline includes 1 to 10 processing stages. When the camera component 10 called by each first virtual pipeline processing stage 41 reaches the second or third, the first to second camera components 10 are in an idle state. Then the processing stage of the second virtual pipeline can be called starting from the first idle camera component 10 to execute the second request instruction. It can be understood that for every idle camera component 10 called by the first virtual pipeline processing stage 41, the second virtual pipeline processing stage 42 can call the idle camera component 10.

[0103] Compared with the prior art, in which the next virtual pipeline can only perform corresponding operations after all processing stages of each virtual pipeline are completed (for example, the second virtual pipeline can only start processing from the first processing stage after all 10 processing stages of the first virtual pipeline are completed and the camera component 10 is called), the present application provides a solution in which multiple virtual components call multiple camera components 10 in parallel. For example, as long as one of the camera components 10 called in the processing stage of the first virtual pipeline is idle, the second virtual pipeline can start executing operations and call the corresponding idle camera component 10, thereby greatly improving the utilization efficiency of the camera component 10 and saving processing time.

[0104] like Figure 9 As shown, any of the first to third embodiments provided in this application may further include the following steps:

[0105] S8: Output the last frame image of the first virtual pipeline and the first frame image of the second virtual pipeline in sequence.

[0106] Specifically, such as Figure 4As shown, the first virtual pipeline has multiple processing stages, each outputting a corresponding image; the second virtual pipeline has multiple processing stages, each outputting a corresponding image. The last frame of the first virtual pipeline outputs the first frame of the second virtual pipeline in sequence, seamlessly connecting to each other and avoiding image interruptions or frame drops. For example, if the first virtual pipeline has three processing stages, after the first processing stage of the first virtual pipeline completes, the second virtual pipeline can begin processing, starting with the first camera component that has not been called by the first virtual pipeline. At this time, the second and third processing stages of the first virtual pipeline continue to execute. The processing stages of the second virtual pipeline operate sequentially. After all three processing stages of the first virtual pipeline are completed, they seamlessly connect to the first stage of the second virtual pipeline, thus eliminating image interruptions or frame drops. This continues until all processing stages of the second virtual pipeline are completed. After the first virtual pipeline is idle, it can continue to respond to new application calls.

[0107] The method for controlling multiple camera components disclosed in this application includes: receiving a first request instruction from a first application to call multiple camera components; and establishing and executing a first virtual pipeline in response to the first request instruction, wherein the first virtual pipeline includes multiple first virtual pipeline processing stages, and each of the multiple first virtual pipeline processing stages calls one of the multiple camera components to execute the first request instruction. The method for controlling multiple camera components provided in this application enables multiple components of the same camera to be called by multiple applications, and when multiple applications call multiple camera components simultaneously, there will be no image interruption or obvious frame drop issues.

[0108] See also Figure 13 , Figure 13 It is a structural diagram of a computer device provided in one embodiment of the present application.

[0109] The computer device 200 may specifically include a processor 210 and a memory 220 . The memory 220 is coupled to the processor 210 .

[0110] Processor 210 is used to control the operation of computer device 200. Processor 210 may also be referred to as a CPU (Central Processing Unit). Processor 210 may be an integrated circuit chip with signal processing capabilities. Processor 210 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. A general-purpose processor may be a microprocessor, or processor 210 may be any conventional processor.

[0111] The memory 220 is used to store computer programs and can be RAM, ROM, or other types of storage devices. Specifically, the memory may include one or more computer-readable storage media, which may be non-transitory. The memory may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory is used to store at least one program code.

[0112] The processor 210 is configured to execute a computer program stored in the memory 220 to implement the camera assembly described in the embodiment of the camera assembly of the present application.

[0113] In some embodiments, the computer device 200 may further include a peripheral device interface 230 and at least one peripheral device. The processor 210, memory 220, and peripheral device interface 230 may be connected via a bus or signal lines. Each peripheral device may be connected to the peripheral device interface 230 via a bus, signal lines, or circuit boards. Specifically, the peripheral device may include at least one of a radio frequency circuit 240, a display screen 250, an audio circuit 260, and a power supply 270.

[0114] The peripheral device interface 230 can be used to connect at least one I / O (Input / Output)-related peripheral device to the processor 210 and the memory 220. In some embodiments, the processor 210, the memory 220, and the peripheral device interface 230 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 210, the memory 220, and the peripheral device interface 230 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0115] The RF circuit 240 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 240 communicates with communication networks and other communication devices via electromagnetic signals and serves as the communication circuitry for the computer device 200. The RF circuit 240 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the RF circuit 240 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The RF circuit 240 can communicate with other terminals via at least one wireless communication protocol. Such wireless communication protocols include, but are not limited to, the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 240 may also include circuitry related to Near Field Communication (NFC), although this application does not limit this.

[0116] The display screen 250 is used to display a user interface (UI). This UI may include graphics, text, icons, videos, or any combination thereof. When the display screen 250 is a touch screen display, it is also capable of collecting touch signals on or above the surface of the display screen 250. These touch signals can be input as control signals to the processor 210 for processing. In this case, the display screen 250 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there can be a single display screen 250, located on the front panel of the computer device 200. In other embodiments, there can be at least two display screens 250, located on different surfaces of the computer device 200 or in a foldable design. In still other embodiments, the display screen 250 can be a flexible display screen, located on a curved or foldable surface of the computer device 200. Furthermore, the display screen 250 can be configured as a non-rectangular irregular shape, i.e., a special-shaped screen. The display screen 250 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0117] The audio circuit 260 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals to be input into the processor 210 for processing, or input into the radio frequency circuit 240 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there can be multiple microphones, which are respectively arranged in different parts of the computer device 200. The microphone can also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 210 or the radio frequency circuit 240 into sound waves. The speaker can be a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 260 may also include a headphone jack.

[0118] Power supply 270 is used to power various components in computer device 200. Power supply 270 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 270 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0119] For a detailed description of the functions and execution processes of the various functional modules or components in the embodiment of the computer device 200 of the present application, please refer to the description in the above-mentioned embodiment of the camera component of the present application, and no further details will be given here.

[0120] In the several embodiments provided in this application, it should be understood that the disclosed computer device 200 and camera assembly can be implemented in other ways. For example, the various embodiments of the computer device 200 described above are merely illustrative. For example, the division of modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0121] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0122] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0123] See also Figure 14 , Figure 14 This is a schematic block diagram of the structure of a computer-readable storage medium provided in one embodiment of the present application.

[0124] See Figure 14 , if the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium 300. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions / computer programs to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present invention. The aforementioned storage medium includes various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, as well as computer devices such as computers, mobile phones, laptops, tablet computers, cameras, etc. having the above-mentioned storage medium.

[0125] The description of the execution process of the program data in the computer-readable storage medium 300 can be made with reference to the description in the embodiment of the camera assembly of the present application, and will not be repeated here.

[0126] The above description is only an implementation method of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of this application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for controlling multiple camera assemblies, characterized in that: The following steps are involved: Receiving a first request instruction from a first application to call the plurality of camera components; as well as In response to the first request instruction, a first virtual pipeline is established and executed, wherein the first virtual pipeline includes multiple first virtual pipeline processing stages, and the multiple first virtual pipeline processing stages respectively call one of the multiple camera components to execute the first request instruction.

2. The control method of multiple camera assemblies according to claim 1, characterized in that: The following steps are also included: Receiving a second request instruction of at least one second application to call the plurality of camera components; wherein the second application is different from the first application; In response to the second request instruction, a second virtual pipeline is established and executed, wherein the second virtual pipeline includes multiple second virtual pipeline processing stages, and the multiple second virtual pipeline processing stages respectively call one of the multiple camera components to execute the second request instruction; wherein, at least one of the multiple second virtual pipeline processing stages is the same as the camera component called by at least one of the multiple first virtual pipeline processing stages.

3. The control method of multiple camera assemblies according to claim 1, characterized in that: The following steps are also included: Receiving a second request instruction of at least one second application to call the plurality of camera components; wherein the second application is different from the first application; In response to the second request instruction, a second virtual pipeline is established and executed, wherein the second virtual pipeline includes multiple second virtual pipeline processing stages, and the multiple second virtual pipeline processing stages respectively call one of the multiple camera components to execute the second request instruction, wherein the multiple second virtual pipeline processing stages are the same as the multiple first virtual pipeline processing stages.

4. The control method of multiple camera assemblies according to claim 2 or 3, characterized in that: The following steps are also included: Creating at least one first physical pipeline corresponding to the multiple camera components; wherein the first physical pipeline includes multiple first physical pipeline processing stages, the multiple first virtual pipeline processing stages correspond to the multiple first physical pipeline processing stages, and the multiple first physical pipeline processing stages respectively correspond to the multiple camera components; Create at least one second physical pipeline corresponding to the multiple camera components; wherein the second physical pipeline includes multiple second physical pipeline processing stages, the multiple second virtual pipeline processing stages correspond to the multiple second physical pipeline processing stages, and the multiple second physical pipeline processing stages respectively correspond to the multiple camera components.

5. The control method of multiple camera assemblies according to claim 4, characterized in that: While executing the second virtual pipeline, unfinished related operations of the first virtual pipeline are executed.

6. The control method of multiple camera assemblies according to claim 5, characterized in that: The step of establishing and executing the second virtual pipeline in response to the second request instruction includes: Switching at least one of the plurality of camera components to execute the second request instruction; The step of executing the unfinished related operations of the first virtual pipeline while executing the second virtual pipeline includes: The remaining components of the multiple camera components corresponding to the multiple first virtual pipeline processing stages continue to perform the unfinished related operations of the first virtual pipeline.

7. The method for controlling multiple camera assemblies according to claim 4, wherein: The first request instruction and the second request instruction come from different applications respectively; The first virtual pipeline and the second virtual pipeline of the multiple applications run independently of each other.

8. The method according to claim 7, characterized in that Before establishing the first virtual pipeline and the second virtual pipeline corresponding to the multiple applications, the method includes: determining whether a plurality of identical camera components corresponding to a plurality of physical pipeline processing stages in the first physical pipeline and the second physical pipeline have support parameters corresponding to each of the applications; In response to the plurality of camera components having support parameters corresponding to each of the applications, the support parameters of the plurality of camera components corresponding to the first virtual pipeline and the second virtual pipeline are the same; In response to the plurality of camera components not having support parameters corresponding to each of the applications, the support parameters of the plurality of camera components corresponding to the first virtual pipeline and the second virtual pipeline are different.

9. The method for controlling multiple camera assemblies according to claim 2 or 3, wherein: Also includes: The last frame image of the first virtual pipeline and the first frame image of the second virtual pipeline are output in sequence.

10. The method according to claim 2 or 3, characterized in that: The plurality of camera components corresponding to the plurality of first physical pipeline processing stages and the plurality of second physical pipeline processing stages include one or more of image source acquisition, image noise reduction, frame capture, and distortion-aware quantization.

11. A computer device, characterized in that: include: processor; a memory connected to the processor, and configured to store a computer program executable on the processor; wherein the processor implements the method according to any one of claims 1 to 10 when executing the computer program; The camera assembly is connected to the processor and is used to respond to commands from the processor and perform corresponding operations.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images