Image generation method and device, electronic equipment and computer readable storage medium

Abstract

The embodiment of the invention discloses an image generation method and device, electronic equipment and a computer readable storage medium, and relates to the equipment scheduling technology, and the method comprises the steps: obtaining the current state information of at least one to-be-selected camera, the current state information comprises the current state of each to-be-selected camera in at least one dimension, and screening at least one target camera from at least one to-be-selected camera according to the current state information, shooting through the at least one target camera to obtain at least one scene image, and combining the scene images to obtain a target image. According to the method, the target camera is screened based on the current state information of each to-be-selected camera, and then the screened camera is controlled to shoot, so that the reasonability of camera scheduling is improved. And the cameras are dispatched based on the current state information of the to-be-selected camera, so that the method is more suitable for complex and dynamic scenes, and the requirements of a use environment can be better met.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of device scheduling, and specifically to an image generation method, apparatus, electronic device, and computer-readable storage medium. Background Art

[0002] Currently, multiple cameras are being considered for collaborative capture of panoramic images or video streams. However, this collaboration requires scheduling technology. Current resource scheduling technologies for multi-camera collaboration primarily rely on traditional scheduling algorithms, such as heuristic algorithms and rule engines.

[0003] In addition, these current scheduling algorithms are generally based on static scheduling and simple priority scheduling, which means that they can only meet basic needs in some simple application scenarios. However, in complex and highly dynamic environments, they have technical problems such as poor rationality and difficulty in meeting needs. Summary of the Invention

[0004] The embodiments of the present application provide an image generation method, device, electronic device, and computer-readable storage medium, which can improve the rationality of camera scheduling and better meet the needs of the usage environment.

[0005] In a first aspect, an embodiment of the present application provides an image generation method, the method comprising:

[0006] Acquire current state information of at least one to-be-selected camera, where the current state information includes a current state of each of the to-be-selected cameras in at least one dimension;

[0007] Filtering at least one target camera from the at least one to-be-selected camera according to the current state information;

[0008] Obtain at least one scene image by photographing with the at least one target camera;

[0009] The scene images are merged to obtain a target image.

[0010] In a second aspect, an embodiment of the present application further provides an image generating device, the device comprising:

[0011] An acquisition module is used to obtain current status information of at least one camera to be selected, wherein the current status information includes the current status of each camera to be selected in at least one dimension;

[0012] A screening module, configured to screen at least one target camera from the at least one camera to be selected according to the current state information;

[0013] a shooting module, configured to obtain at least one scene image by shooting with the at least one target camera;

[0014] The merging module is used to merge the scene images to obtain a target image.

[0015] Optionally, in some embodiments of the present application, screening at least one target camera from the at least one to-be-selected camera according to the current state information includes:

[0016] Obtaining historical status information corresponding to each of the to-be-selected cameras;

[0017] Predicting target state information of each of the candidate cameras at at least one future moment using a state prediction model based on the current state information and the historical state information;

[0018] The target camera is filtered from the at least one to-be-selected camera according to the target state information.

[0019] Optionally, in some embodiments of the present application, screening the target camera from the at least one to-be-selected camera according to the target state information includes:

[0020] Determine the screening target information and set the device selection variables;

[0021] constructing an objective function based on the screening target information and the device selection variables;

[0022] determining constraints for the target state information;

[0023] The objective function is solved according to the constraint condition and the target state information to obtain a device selection result corresponding to the device selection variable, wherein the device selection result indicates the filtered target camera.

[0024] Optionally, in some embodiments of the present application, solving the objective function according to the constraint condition and the target state information to obtain the device selection result corresponding to the device selection variable includes:

[0025] Performing a quadratic programming conversion process on the objective function according to the target state information to obtain an optimization function;

[0026] Solving the optimization function using the constraint conditions and the solver to obtain the device selection result;

[0027] Among them, if the optimization function is solved by the constraint conditions and the solver in time, the target camera is filtered according to the preset state relationship information through a greedy algorithm to obtain the device selection result.

[0028] Optionally, in some embodiments of the present application, the merging of the scene images to obtain the target image includes:

[0029] Extracting feature points from each of the scene images to obtain at least one feature point information;

[0030] Performing matching processing on the feature point information of each of the scene images to obtain a feature point matching result;

[0031] The target image is obtained by merging the scene images according to the feature point matching result and the feature point information.

[0032] Optionally, in some embodiments of the present application, merging the scene images to obtain the target image according to the feature point matching result and the feature point information includes:

[0033] Determining the position information of the target camera using Lie group optimization according to the feature point matching result and the feature point information;

[0034] Based on the pose information, the scene images are merged through bundle adjustment to obtain the target image.

[0035] Optionally, in some embodiments of the present application, after merging the scene images to obtain the target image, the device further includes:

[0036] Obtaining the actual completion time of the target image;

[0037] determining delay deviation information based on the expected completion time and the actual completion time;

[0038] The weight coefficients in the objective function are adjusted according to the delay deviation information, and a new target image is generated based on the adjusted weight coefficients.

[0039] In a third aspect, an embodiment of the present application further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps in the above-mentioned image generation method are implemented.

[0040] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above-mentioned image generation method are implemented.

[0041] In a fifth aspect, embodiments of the present application further provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations described in the embodiments of the present application.

[0042] In summary, the embodiments of the present application obtain current status information of at least one camera to be selected, and the current status information includes the current status of each camera to be selected in at least one dimension. At least one target camera is selected from at least one camera to be selected based on the current status information, and at least one scene image is obtained by photographing the at least one target camera, and the scene images are merged to obtain the target image.

[0043] By selecting target cameras based on their current status information and then controlling them to shoot, the rationality of camera scheduling is improved. Scheduling cameras based on their current status information is also more adaptable to complex and dynamic scenarios and better meets the needs of the usage environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in this application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 Schematic diagram of a scenario in which a terminal device according to an embodiment of the present application executes the image generation method;

[0046] Figure 2 Schematic diagram of the process of generating an image according to an embodiment of the present invention;

[0047] Figure 3 is a structural diagram of an image generating device provided in an embodiment of the present application;

[0048] Figure 4 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0050] The embodiments of the present application provide an image generation method, device, electronic device and computer-readable storage medium. Specifically, the embodiments of the present application provide an image generation device suitable for an electronic device, which includes a terminal device, and the terminal device includes but is not limited to a mobile phone, a tablet computer, a laptop computer, a smart TV and other central control devices. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN, Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms, etc. The server can be directly or indirectly connected via wired or wireless communication.

[0051] For example, see Figure 1 , Figure 1 : is a schematic diagram of a scenario in which a terminal device according to an embodiment of the present application executes the image generation method. Specifically, the specific execution process of the terminal device executing the image generation method is as follows:

[0052] The terminal device 101 obtains the current status information of at least one camera to be selected, and the current status information includes the current status of each camera to be selected in at least one dimension. According to the current status information, at least one target camera is selected from at least one camera to be selected, and at least one scene image is obtained by shooting with the at least one target camera, and the scene images are merged to obtain the target image.

[0053] The terminal device acts as a control center and schedules each camera based on its current state information. For example, the terminal device schedules each camera to capture scene images based on the camera's current state information and merges the scene images into a target image, where the target image includes a panoramic image.

[0054] In summary, the embodiments of the present application improve the rationality of camera scheduling by screening target cameras based on the current status information of each candidate camera and then controlling the screened cameras to shoot. Scheduling each camera based on the current status information of the candidate cameras is also more adaptable to complex and dynamic scenes and better meets the needs of the usage environment.

[0055] It should be noted that the order of description of the following embodiments does not limit the priority order of the embodiments.

[0056] See also Figure 2 , Figure 2 This is a flowchart of an image generation method provided in an embodiment of the present application. Although the flowchart shows a logical order, in some cases, the steps shown or described may be performed in a different order than that shown in the flowchart. Specifically, the execution subject of the image generation method includes a terminal device, and the specific process of the image generation method is as follows:

[0057] 201. Obtain current status information of at least one camera to be selected, where the current status information includes the current status of each camera to be selected in at least one dimension.

[0058] The cameras to be selected are cameras that are arranged or installed in real scenes. These cameras are usually added to a local area network, and the scheduling and control of each camera is realized through the local area network. For example, the response of each camera to be selected is received through the soft bus broadcast discovery protocol, and the device map of each camera to be selected is constructed. Edge weight Represents the spatial proximity of devices, obtained by calculating the control distance of the selected camera, and constructing a connected subgraph And ensure that the communication delay between each candidate camera in the subgraph is ≤50ms. Where D represents the set of candidate cameras {d1,...,dn}, ε represents the connection relationship between the candidate cameras, L i Indicates the spatial coordinates of the camera to be selected.

[0059] Among them, the current status information refers to the current status of the selected camera in various dimensions, reflecting the basic situation of the camera's current device capabilities. For example, dimensions include but are not limited to computing power, available bandwidth, remaining energy (such as remaining power), resolution, or spatial position and orientation.

[0060] 202. Filter at least one target camera from at least one candidate camera according to current state information.

[0061] Specifically, target cameras that meet the requirements are screened from multiple candidate cameras based on their current status information. For example, a subset of target cameras that meet the requirements can be screened from the multiple candidate cameras based on their remaining battery life, bandwidth, and spatial location and orientation. The target cameras are the selected candidate cameras.

[0062] 203. Obtain at least one scene image by photographing with at least one target camera.

[0063] The scene image is an image obtained by photographing the on-site scene. For example, the scene image is obtained by controlling each target camera to photograph the scene in which it is located.

[0064] 204. Merge the scene images to obtain a target image.

[0065] The target image is an image that meets the requirements. For example, the target image includes but is not limited to a panoramic image. For example, a panoramic image is obtained by stitching together scene images from different perspectives. Accordingly, in an embodiment of the present application, after obtaining panoramic images at various time points, the panoramic images at various time points can be synthesized into a panoramic video stream.

[0066] In summary, the embodiments of the present application improve the rationality of camera scheduling by screening target cameras based on the current status information of each candidate camera and then controlling the screened cameras to shoot. Scheduling each camera based on the current status information of the candidate cameras is also more adaptable to complex and dynamic scenes and better meets the needs of the usage environment.

[0067] It is understandable that the purpose of camera scheduling is to respond to upcoming tasks, such as a panoramic photography task. Therefore, future device status information can further improve the accuracy of camera scheduling. Therefore, the embodiments of the present application predict the target state information of each candidate camera in the future and use the target state information to control the scheduling of each candidate camera. That is, optionally, in some embodiments of the present application, the step of "selecting at least one target camera from at least one candidate camera based on the current state information" includes:

[0068] Obtain the historical status information corresponding to each camera to be selected;

[0069] Based on the current state information and historical state information, the state prediction model is used to predict the target state information of each candidate camera at least at a future moment;

[0070] A target camera is filtered from at least one candidate camera according to the target state information.

[0071] The historical status information is the status information of the camera to be selected in the historical stage, such as the status information before the current moment.

[0072] Among them, the state prediction model is used to predict the state information of the future moment based on the state information of the selected camera in the current and historical stages. The state prediction model module is a pre-trained deep learning model or machine learning model. In addition, the state prediction model can also be an estimation algorithm. For example, the state prediction model includes a Kalman filter (Kalman Filter), wherein the Kalman filter (Kalman Filter) is an optimal estimation algorithm based on Bayesian probability and linear system state space model.

[0073] For example, the state prediction model is expressed as:

[0074] The input is: the current state information + historical state information obtained by the device at the current time (e.g., computing power, bandwidth, power, etc.), the output is: target status information A and B are state transfer matrices, u i is the control input vector, is the process noise.

[0075] For example, the dimensions corresponding to the status information include computing power Available bandwidth Remaining energy (such as remaining battery power) Resolution R i =[r i1 ,r i2 ] or spatial position and orientation L i =[θ i ,φ i ,x i ,y i ,z i ] etc. Therefore, camera d i The state vector corresponding to the state information is expressed as: i =[C i ,B i ,E i ,R i ,L i ] T , and the tensor corresponding to the global resource status information containing historical status information is expressed as: Where k is the length of the historical state window, defined as: in, represents the current state information and historical state information of the first candidate camera at the current time t. t is the offline time step. For example, the state information of the candidate camera is collected every 100ms, and the state information of the candidate camera 100ms later is predicted.

[0076] It can be understood that this solution of predicting target state information at future moments and scheduling cameras based on the target state information can perform dynamic scheduling based on a real-time changing dynamic environment. For example, if the camera's power is predicted to decline, it will switch to other devices in advance. If high-load task requirements are predicted, bandwidth will be allocated in advance to reserve bandwidth for panoramic shooting tasks. If the computing power of the selected camera is predicted to decrease (such as a computing power decrease of more than 20%), it will switch to scheduling other cameras, etc.

[0077] In some embodiments of the present application, a screening target can be set, and an objective function can be constructed based on the screening target. The device selection result for the target camera can be obtained by solving the objective function. In addition, a method combining mixed integer nonlinear programming (MINLP) and receding horizon control can be used to solve the objective function. That is, optionally, in some embodiments of the present application, the step of "screening the target camera from at least one candidate camera based on the target state information" includes:

[0078] Determine the screening target information and set the device selection variables;

[0079] Constructing an objective function based on the screening target information and device selection variables;

[0080] Determine constraints on target state information;

[0081] The objective function is solved according to the constraint conditions and the target state information to obtain a device selection result corresponding to the device selection variable, wherein the device selection result indicates the screened target camera.

[0082] Among them, the target camera is screened based on the screening target information, which improves the rationality of the target camera screening.

[0083] The screening target information refers to the screening target, that is, the requirements that the camera screening is expected to meet. For example, the screening target information includes minimizing the shooting task completion time T, minimizing the total energy consumption E, and minimizing the view coverage deviation ΔV, where the view coverage deviation includes but is not limited to the difference between the actual field of view of the camera and the expected or nominal field of view;

[0084] The device selection variable is represented by X=[x1,x2,...,x n ], where n represents the number of cameras to be selected, and x1∈{0,1}, where 0 represents not selecting and 1 represents selecting, i.e., the camera to be selected represented by 1 is used as the target camera.

[0085] And, the objective function is expressed as: min X(αT+βE+γΔV), where α, β, and γ are weight coefficients that are dynamically adjusted according to the task type, i.e., Among them, λ T ,λ E ,λ V is the task priority parameter, j corresponds to the task type, and in the embodiment of the present application, it is the priority parameter corresponding to the panoramic shooting task, wherein each task is mapped to the priority parameter, and there is no specific restriction on the priority parameter corresponding to each task.

[0086] Accordingly, constraints are used to solve the objective function. The constraints include resolution constraints, bandwidth constraints, or spatial coverage constraints. For example, the resolution constraint is expressed as: Among them, W req is the width threshold, H req is the height threshold. The bandwidth constraint is expressed as: The spatial coverage constraint is expressed as: in is the camera frustum function, Target coverage area.

[0087] In this embodiment, the objective function is solved using the target state information corresponding to the time window [t, t+h] to obtain the device selection result. The device selection result is the optimal solution corresponding to the current selection scheme. For example, the device selection result is represented by X = [0, 1, ..., 1]. Similarly, when scrolling to the next time window [t+1, t+h+1], the optimization is repeated to determine the optimal solution for the device selection result corresponding to t+1.

[0088] In the embodiments of the present application, a mixed integer quadratic programming (MIQP) suitable for scenarios with high real-time requirements may also be used to solve the device selection variables. That is, optionally, in some embodiments of the present application, the step of "solving the objective function according to the constraints and the target state information to obtain the device selection result corresponding to the device selection variable" includes:

[0089] According to the target state information, the target function is converted into an optimization function by performing quadratic programming;

[0090] The optimization function is solved by using constraints and solvers to obtain the equipment selection result;

[0091] Among them, if the optimization function is solved by the constraints and solver in timeout, the target camera is filtered according to the preset state relationship information through the greedy algorithm to obtain the device selection result.

[0092] Among them, MIQP is an efficient approximate solution for MINLP, which has a higher effect in scenarios with high real-time requirements and helps to reduce the computational complexity. For example, the objective function is changed from min X (αT+βE+γΔV) is converted to: Where C is the camera capability matrix, for example, C ij Indicates camera d i The demand for task j is represented by a vector, and D is the corresponding demand for task j. In the embodiment of the present application, the equipment selection result is obtained by solving the problem using Gurobi or CPLEX solver.

[0093] Among them, in the embodiment of the present application, if the optimization function is solved by the constraint condition and the solver, the preset state information (such as the computing power and the remaining energy ratio) is preferentially selected by the greedy algorithm. )The largest camera.

[0094] In the embodiments of the present application, the merging of the scene images is achieved by matching feature points. That is, optionally, in some embodiments of the present application, the step of "merging the scene images to obtain the target image" includes:

[0095] Extracting feature points from each scene image to obtain at least one feature point information;

[0096] Perform matching processing on the feature point information of each scene image to obtain feature point matching results;

[0097] Based on the feature point matching results and feature point information, the scene images are merged to obtain the target image.

[0098] Feature point information, also known as image feature points (also called key points or points of interest), refers to pixels in an image with significant local characteristics. These points remain stable under different viewing angles, lighting, or deformation, and can be used for tasks such as image matching, object recognition, and 3D reconstruction. The SIFT algorithm is used to extract feature point information from scene images.

[0099] Accordingly, by performing feature point matching, matching point pairs can be identified. These matching point pairs are two representations of the same key point in different images. Based on these matching point pairs, the images can then be aligned to obtain the target image. For example, by merging the images of different scenes into a single panoramic image based on the same portion of the scene involved.

[0100] In the embodiments of the present application, the camera pose can be used as a reference for image alignment to achieve merging of scene images. That is, optionally, in some embodiments of the present application, the step of "merging the scene images to obtain the target image based on the feature point matching results and feature point information" includes:

[0101] Determine the position information of the target camera using Lie group optimization based on the feature point matching results and feature point information;

[0102] Based on the pose information, the target image is obtained by merging the scene images through bundle adjustment.

[0103] For example, the initial pose, focal length and other parameters of each target camera are stored in the perspective parameter matrix M, and the initial pose is converted into Lie algebra parameters through feature point matching results, feature point information, and Lie group optimization. Local fine-tuning is performed to obtain the optimized pose information, and then the pose information of each camera is jointly optimized through bundle adjustment. According to the optimized camera pose information, each scene image is projected into the same panoramic coordinate system to obtain the target image.

[0104] For example, the viewing parameter matrix M is expressed as:

[0105]

[0106] Among them, θ i is the azimuth angle, φi is the elevation angle, i is 1-m, f i is the focal length, (u 0i ,v 0i ) is the main point coordinate (such as camera coordinate), t m is the timestamp.

[0107] Among them, the objective function corresponding to Lie group optimization is: Among them, ξ is the Lie algebra parameter vector, T∈SE(3) is the camera pose transformation matrix, and π(·) is the projection function, such as π(T(ξ)·p j ) represents the feature point information p j The projection, p j and q j The two feature point information that constitute the matching point pair.

[0108] Among them, the objective function corresponding to bundle adjustment is: Among them, ρ is the Huber loss function, which is used to suppress the influence of outliers, π(·) is the projection function, and T i is the pose of the i-th camera (external reference), p j represents the homogeneous coordinates of the j-th three-dimensional space point (map point), q ij Represents the j-th 3D point p j The observed 2D pixel coordinates in the i-th camera image.

[0109] Furthermore, in embodiments of the present application, the quality of the target avatar generation task may be evaluated, and the objective function may be adjusted based on the evaluation results to improve the rationality of camera scheduling. That is, optionally, in some embodiments of the present application, after the step of "merging the scene images to obtain the target image", the method further includes:

[0110] Get the actual completion time of the target image;

[0111] Determine delay deviation information based on expected completion time and actual completion time;

[0112] The weight coefficients in the objective function are adjusted according to the delay deviation information, and a new target image is generated based on the adjusted weight coefficients.

[0113] The actual completion time is obtained through measurement, the expected completion time is the predetermined expected completion time, and the delay deviation information is the time difference between the expected completion time and the actual completion time. For example, in this embodiment of the present application, if the absolute value of the time difference exceeds 10% of the expected completion time, then the weight coefficients α, β, and γ in the objective function are adjusted.

[0114] The specific adjustment can be made through the control quantity, which can be calculated based on the time difference. The control quantity u(t) is expressed as:

[0115]

[0116] Among them, e(t) represents the time difference, that is, the delay deviation information; K p Represents the proportionality coefficient, K i Indicates the integral coefficient, K d represents the differential coefficient. Correspondingly, the adjusted weight coefficients can be expressed as: α′=α·(1+0.5u(t)), β′=β·(1-0.3u(t)), γ′=γ·(1-0.2u(t)).

[0117] In summary, the embodiments of the present application improve the rationality of camera scheduling by screening target cameras based on the current status information of each candidate camera and then controlling the screened cameras to shoot. Scheduling each camera based on the current status information of the candidate cameras is also more adaptable to complex and dynamic scenes and better meets the needs of the usage environment.

[0118] To facilitate better implementation of the image generation method of the present application, the present application also provides an image generation device based on the above image generation method. The meanings of the terms herein are the same as those in the above image generation method, and the specific implementation details can be referred to the description in the method embodiment.

[0119] See also Figure 3 , Figure 3 : is a schematic diagram of the structure of an image generating device provided in an embodiment of the present application, wherein the image generating device may be specifically as follows:

[0120] An acquisition module 301 is configured to acquire current state information of at least one camera to be selected, where the current state information includes the current state of each camera to be selected in at least one dimension;

[0121] A screening module 302 is configured to screen at least one target camera from at least one candidate camera according to current state information;

[0122] A shooting module 303 is configured to obtain at least one scene image by shooting with at least one target camera;

[0123] The merging module 304 is configured to merge the scene images to obtain a target image.

[0124] Optionally, in some embodiments of the present application, screening at least one target camera from at least one candidate camera according to current state information includes:

[0125] Obtain the historical status information corresponding to each camera to be selected;

[0126] Based on the current state information and historical state information, the state prediction model is used to predict the target state information of each candidate camera at least at a future moment;

[0127] A target camera is filtered from at least one candidate camera according to the target state information.

[0128] Optionally, in some embodiments of the present application, screening a target camera from at least one candidate camera according to the target state information includes:

[0129] Determine the screening target information and set the device selection variables;

[0130] Constructing an objective function based on the screening target information and device selection variables;

[0131] Determine constraints on target state information;

[0132] The objective function is solved according to the constraint conditions and the target state information to obtain a device selection result corresponding to the device selection variable, wherein the device selection result indicates the screened target camera.

[0133] Optionally, in some embodiments of the present application, solving the objective function according to the constraint conditions and the target state information to obtain a device selection result corresponding to the device selection variable includes:

[0134] According to the target state information, the target function is converted into an optimization function by performing quadratic programming;

[0135] The optimization function is solved by using constraints and solvers to obtain the equipment selection result;

[0136] Among them, if the optimization function is solved by the constraints and solver in timeout, the target camera is filtered according to the preset state relationship information through the greedy algorithm to obtain the device selection result.

[0137] Optionally, in some embodiments of the present application, merging the scene images to obtain the target image includes:

[0138] Extracting feature points from each scene image to obtain at least one feature point information;

[0139] Perform matching processing on the feature point information of each scene image to obtain feature point matching results;

[0140] Based on the feature point matching results and feature point information, the scene images are merged to obtain the target image.

[0141] Optionally, in some embodiments of the present application, merging the scene images to obtain the target image based on the feature point matching results and the feature point information includes:

[0142] Determine the position information of the target camera using Lie group optimization based on the feature point matching results and feature point information;

[0143] Based on the pose information, the target image is obtained by merging the scene images through bundle adjustment.

[0144] Optionally, in some embodiments of the present application, after merging the scene images to obtain the target image, the apparatus further includes:

[0145] Get the actual completion time of the target image;

[0146] Determine delay deviation information based on expected completion time and actual completion time;

[0147] The weight coefficients in the objective function are adjusted according to the delay deviation information, and a new target image is generated based on the adjusted weight coefficients.

[0148] In the embodiment of the present application, the acquisition module 301 first obtains the current status information of at least one camera to be selected, and the current status information includes the current status of each camera to be selected in at least one dimension. The screening module 302 screens at least one target camera from at least one camera to be selected based on the current status information. The shooting module 303 obtains at least one scene image by shooting at least one target camera, and the merging module 304 merges the scene images to obtain the target image.

[0149] In this embodiment of the present application, by selecting target cameras based on the current status information of each candidate camera, and then controlling the selected cameras to shoot, the rationality of camera scheduling is improved. Scheduling each camera based on the current status information of the candidate cameras is also more adaptable to complex and dynamic scenes and better meets the needs of the usage environment.

[0150] In addition, the present application also provides an electronic device, such as Figure 4 As shown, it shows a schematic diagram of the structure of the electronic device involved in this application, specifically:

[0151] The electronic device may include one or more processing core processors 401, one or more computer-readable storage media memories 402, a power supply 403, an input unit 404 and other components. Those skilled in the art will understand that Figure 4 The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently.

[0152] Processor 401 is the control center of the electronic device, connecting the various parts of the entire electronic device using various interfaces and lines. By running or executing software programs and / or modules stored in memory 402 and accessing data stored in memory 402, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, processor 401 may include one or more processing cores; preferably, processor 401 may integrate an application processor and a modem processor, wherein the application processor primarily processes the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 401.

[0153] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0154] The electronic device also includes a power supply 403 for supplying power to various components. Preferably, the power supply 403 can be logically connected to the processor 401 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The power supply 403 can also include one or more DC or AC power supplies, a recharging system, a power supply device debugging circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0155] The electronic device may further include an input unit 404, which may be configured to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0156] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the electronic device loads the executable files corresponding to one or more application processes into the memory 402 according to the following instructions, and the processor 401 runs the application stored in the memory 402, thereby implementing the steps of any of the image generation methods provided in the embodiments of the present application.

[0157] An embodiment of the present application obtains current status information of at least one camera to be selected, and the current status information includes the current status of each camera to be selected in at least one dimension. At least one target camera is selected from the at least one camera to be selected based on the current status information, and at least one scene image is obtained by photographing the at least one target camera, and the scene images are merged to obtain a target image.

[0158] By selecting target cameras based on their current status information and then controlling them to shoot, the rationality of camera scheduling is improved. Scheduling cameras based on their current status information is also more adaptable to complex and dynamic scenarios and better meets the needs of the usage environment.

[0159] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0160] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be accomplished by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0161] To this end, the present application provides a computer-readable storage medium, on which a computer program is stored. The computer program can be loaded by a processor to execute the steps in any one of the image generation methods provided in the present application.

[0162] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0163] The computer-readable storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0164] Since the instructions stored in the computer-readable storage medium can execute the steps in any image generation method provided in the present application, the beneficial effects that can be achieved by any image generation method provided in the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0165] The above is a detailed introduction to an image generation method, device, electronic device and computer-readable storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. An image generation method, characterized in that: The method comprises: Acquire current state information of at least one to-be-selected camera, where the current state information includes a current state of each of the to-be-selected cameras in at least one dimension; Filtering at least one target camera from the at least one to-be-selected camera according to the current state information; Obtain at least one scene image by photographing with the at least one target camera; The scene images are merged to obtain a target image.

2. The image generation method according to claim 1, wherein: The selecting at least one target camera from the at least one to-be-selected camera according to the current state information includes: Obtaining historical status information corresponding to each of the to-be-selected cameras; Predicting target state information of each of the candidate cameras at at least one future moment using a state prediction model based on the current state information and the historical state information; The target camera is filtered from the at least one to-be-selected camera according to the target state information.

3. The image generation method according to claim 2, wherein: The step of screening the target camera from the at least one to-be-selected camera according to the target state information includes: Determine the screening target information and set the device selection variables; constructing an objective function based on the screening target information and the device selection variables; determining constraints for the target state information; The objective function is solved according to the constraint condition and the target state information to obtain a device selection result corresponding to the device selection variable, wherein the device selection result indicates the filtered target camera.

4. The image generation method according to claim 3, wherein: Solving the objective function according to the constraint condition and the target state information to obtain a device selection result corresponding to the device selection variable includes: Performing a quadratic programming conversion process on the objective function according to the target state information to obtain an optimization function; Solving the optimization function using the constraint conditions and the solver to obtain the device selection result; Among them, if the optimization function is solved by the constraint conditions and the solver in time, the target camera is filtered according to the preset state relationship information through a greedy algorithm to obtain the device selection result.

5. The image generation method according to claim 1, wherein: The merging of the scene images to obtain a target image includes: Extracting feature points from each of the scene images to obtain at least one feature point information; Performing matching processing on the feature point information of each of the scene images to obtain a feature point matching result; The target image is obtained by merging the scene images according to the feature point matching result and the feature point information.

6. The image generation method according to claim 5, characterized in that The step of merging the scene images to obtain the target image according to the feature point matching result and the feature point information includes: Determining the position information of the target camera using Lie group optimization according to the feature point matching result and the feature point information; Based on the pose information, the scene images are merged through bundle adjustment to obtain the target image.

7. The image generation method according to claim 3, wherein: After merging the scene images to obtain the target image, the method further includes: Obtaining the actual completion time of the target image; determining delay deviation information based on the expected completion time and the actual completion time; The weight coefficients in the objective function are adjusted according to the delay deviation information, and a new target image is generated based on the adjusted weight coefficients.

8. An image generating device, characterized in that: The device comprises: An acquisition module is used to obtain current status information of at least one camera to be selected, wherein the current status information includes the current status of each camera to be selected in at least one dimension; A screening module, configured to screen at least one target camera from the at least one camera to be selected according to the current state information; a shooting module, configured to obtain at least one scene image by shooting with the at least one target camera; The merging module is used to merge the scene images to obtain a target image.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the image generation method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the image generation method according to any one of claims 1 to 7 are implemented.

Images