Imaging device, display apparatus, and imaging system
By introducing processing circuits in the shooting device and display device, enabling them to move around the subject, capture and update the virtual background, the problems of live shooting of amateur performers and live audience experience are solved, and flexible, low-cost and efficient virtualization is achieved. Background shot.
Patent Information
- Application Number
- CN202310595588.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-24
- Publication Date
- 2024-11-26
AI Technical Summary
The existing virtual production system cannot be adapted to amateur performers to conduct live shooting in different locations, and the green screen background cannot be seen by the audience, which affects the experience.
Provide a shooting device and a display device, through a processing circuit, the shooting unit and the display device can move around the shooting object, capture images of the shooting object and the virtual background. The virtual background is dynamically updated based on the position and posture of the shooting device, and realizes a wider application. Scene and simpler image post-processing.
It realizes the flexibility of live shooting in different locations and the virtual background visible to the live audience, reducing costs and footprint while improving the shooting experience.
Smart Images

Figure CN119031087A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of information processing technology, and in particular to a photographing device, a display device, and a photographing system. Background Art
[0002] Existing virtual production systems are often designed for studio shooting scenes. A green screen virtual live broadcast room is built in a professional studio. Performers (such as Coser or cosplayer (role playing)) are located between the camera and the curtain (such as a green screen). The camera captures the performer's performance, and then through post-production, the green screen is cut out and the background is changed to form a performance video with a preset background, which is then broadcast live.
[0003] The disadvantages of the above virtual production system are: on the one hand, it is usually a professional shooting location, which cannot adapt to the needs of some performers (such as amateur performers) who want to shoot live broadcasts in different locations. On the other hand, the green screen background needs to be replaced with the target background through post-production cutout, which is not enjoyable for the audience watching on site. In many scenarios, while the performers are shooting live broadcasts, there will be live audiences at the performance venue, but the live audiences cannot see the real-time picture, which seriously affects the experience. Summary of the invention
[0004] A brief overview of the present invention is provided below in order to provide a basic understanding of certain aspects of the present invention. It should be understood that this overview is not an exhaustive overview of the present invention. It is not intended to identify key or important parts of the present invention, nor is it intended to limit the scope of the present invention. Its purpose is merely to present certain concepts in a simplified form as a prelude to a more detailed description discussed later.
[0005] According to one aspect of the present disclosure, a shooting device is provided, which includes a processing circuit, and the processing circuit is configured to control so that a shooting unit included in the shooting device moves around a shooting object to capture at least one image including the shooting object and a virtual background picture displayed on a display device, wherein the shooting object is located between the shooting device and the display device, and the display device is set on a first mobile device and can be moved by the first mobile device, so that the virtual background picture has at least one of a predetermined angle, size and position in the picture captured by the shooting device, and the virtual background picture is dynamically updated based on the position and / or posture of the shooting device.
[0006] The shooting device according to the embodiment of the present disclosure can achieve at least one of the following effects: having a wider range of application scenarios, simpler image post-processing, and lower cost.
[0007] According to another aspect of the present disclosure, a display device is provided, which includes a processing circuit, which is configured to control so that a display unit included in the display device displays a virtual background picture, so that a shooting device that can move around a shooting object can capture at least one image including the shooting object and the virtual background picture, wherein the shooting object is located between the shooting device and the display device, and the display device is set on a first mobile device so that it can be moved by the first mobile device, so that the virtual background picture has at least one of a predetermined angle, size and position in the picture captured by the shooting device, and the virtual background picture is dynamically updated based on the position and / or posture of the shooting device.
[0008] The display device according to the embodiment of the present disclosure can achieve at least one of the following effects: having a wider range of application scenarios, simpler image post-processing, lower cost, and smaller footprint.
[0009] According to another aspect of the present disclosure, a shooting system is provided, which includes: a display device, which displays a virtual background picture, and a shooting device, which can move around a shooting object to capture at least one image including the shooting object and the virtual background picture, wherein the shooting object is located between the shooting device and the display device, and the display device is set on a first mobile device and can be moved by the first mobile device so that the virtual background picture has at least one of a predetermined angle, size and position in the picture captured by the shooting device, and the virtual background picture is dynamically updated based on the position and / or posture of the shooting device.
[0010] The shooting system according to the embodiment of the present disclosure can achieve at least one of the following effects: having a wider range of application scenarios, simpler image post-processing, lower cost, and smaller footprint. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] In order to further illustrate the above and other advantages and features of the present invention, the specific embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. The accompanying drawings together with the following detailed description are included in this specification and form a part of this specification. Elements with the same function and structure are represented by the same reference numerals. It should be understood that these drawings only describe typical examples of the present invention and should not be regarded as limiting the scope of the present invention. In the drawings:
[0012] Figure 1 A functional module block diagram of a photographing device according to an embodiment of the present disclosure is shown.
[0013] Figure 2A and 2B 2 is a diagram showing a photographing system from different directions according to an embodiment of the present disclosure.
[0014] Figure 3 FIG. 4 is a flowchart showing an example of mapping and obtaining a three-dimensional model of a subject during a photographing process according to an embodiment of the present disclosure.
[0015] Figure 4 Detailed description is a flowchart showing an example of controlling a screen of a display device when the posture of a photographing apparatus changes according to an embodiment of the present disclosure.
[0016] Figure 5 Detailed description is a flowchart showing an example of controlling movement of a first mobile device when the posture of a photographing device changes according to an embodiment of the present disclosure.
[0017] Figure 6 A schematic diagram showing a shooting center line of a shooting device according to an embodiment of the present disclosure is shown.
[0018] Figure 7 A schematic diagram showing that a shooting center line of a shooting device according to an embodiment of the present disclosure is parallel to a center line of an end component.
[0019] Figure 8 is an example flow chart illustrating the control of the motion of a robotic arm according to an embodiment of the present disclosure.
[0020] Fig. 9 A schematic diagram showing a mechanical arm being fixed to a second mobile device according to an embodiment of the present disclosure is shown.
[0021] Fig.10 is an example flow chart illustrating controlling the movement of a robotic arm and a second mobile device according to an embodiment of the present disclosure.
[0022] Fig.11 A functional module block diagram of a display device according to an embodiment of the present disclosure is shown.
[0023] Fig.12 is a schematic diagram illustrating an example of a display device according to an embodiment of the present disclosure.
[0024] Fig.13 is a schematic diagram showing a connection component according to an embodiment of the present disclosure.
[0025] Fig.14 is a schematic diagram showing a spherical wheel according to an embodiment of the present disclosure.
[0026] Fig.15 An example of a shadow of a photographic subject appearing on a display surface of a display device due to lighting components is shown.
[0027] Fig.16 : is a block diagram showing an example structure of a personal computer that can be employed in the embodiments of the present disclosure. DETAILED DESCRIPTION
[0028] Exemplary embodiments of the present disclosure will be described below in conjunction with the accompanying drawings. For the sake of clarity and conciseness, not all features of the actual implementation are described in the specification. However, it should be understood that many implementation-specific decisions must be made in the process of developing any such actual implementation in order to achieve the developer's specific goals, such as meeting those constraints related to the system and business, and these constraints may vary from implementation to implementation. In addition, it should be understood that although the development work may be very complex and time-consuming, it is only a routine task for those skilled in the art who benefit from the content of this disclosure.
[0029] It is also necessary to explain here that in order to avoid obscuring the present disclosure due to unnecessary details, only the device structure and / or processing steps closely related to the scheme according to the present disclosure are shown in the accompanying drawings, while other details that are not closely related to the present disclosure are omitted.
[0030] The embodiments according to the present disclosure are described in detail below with reference to the accompanying drawings.
[0031] Figure 1 FIG. 1 shows a functional module block diagram of a photographing device 100 according to an embodiment of the present disclosure. Figure 1 As shown, the shooting device 100 includes: a shooting control unit 101, which can be configured to control the shooting device, and a shooting processing unit 102, which can be configured to enable the shooting unit included in the shooting device to move around the shooting object to capture at least one image including the shooting object and a virtual background picture displayed on the display device.
[0032] The shooting control unit 101 and the shooting processing unit 102 may be implemented by one or more processing circuits, which may be implemented as a chip or a processor, for example. Figure 1 The various functional units shown in the figure are merely logical modules divided according to the specific functions they implement, and are not used to limit the specific implementation method.
[0033] The photographing apparatus 100 may be implemented at a chip level or at a device level.
[0034] The photographed object (also referred to as the target object, the photographed person, or the subject) is located between the photographing device 100 and the display device; the display device is disposed on the first mobile device and can be moved by the first mobile device, so that the virtual background image has at least one of a predetermined angle, size, and position in the image captured by the photographing device 100, and the virtual background image is dynamically updated based on the position and / or posture of the photographing device 100. For example, the first mobile device can not only drive the display device to translate along the ground direction, but also drive the display device to rise and fall vertically to the ground direction.
[0035] Figure 2A and 2B 1 is a diagram showing a shooting system shown from different directions according to an embodiment of the present disclosure. The shooting system includes a shooting device 100, a display device, and a first mobile device. The shooting system is, for example, a movable automated virtual production system. Figure 2A The front side of the display device is shown, and for simplicity, the first mobile device is not shown. Figure 2B A first mobile device is shown in FIG.
[0036] The photographing device 100 is, for example, a camera, which is, for example, a digital camera, which has at least one of the functions of an RGB sensor, a depth sensor, a ToF (time of flight) sensor, a 6-axis acceleration sensor (i.e., an acceleration sensor and a gyroscope), and, for example, has a communication function that can send information to the outside in a wireless form and can receive information from the outside. The photographing device 100 is, for example, other devices that can realize the above functions, which will not be repeated here. In the following, the photographing device 100 is sometimes referred to as a camera. The camera can be used to track and shoot the target object.
[0037] The position of the photographing device 100 is, for example, the position of the center point of the camera lens in space; the posture of the photographing device 100 is, for example, the angle of the camera orientation. In a three-dimensional coordinate system, the position is, for example, the moving position along the three rectangular coordinate axes of x, y, and z, which can be determined by three coordinate values, such as (x, y, z). The posture can be determined by three angle values, which are the angles of rotation around the x, y, and z axes, respectively. These three angles can be respectively called: roll angle (roll), pitch angle (pitch), and yaw angle (yaw). In computer representation, "position and posture" refers to a set of 6 ordered numbers. For example, (10, 0, 200, 180, 90, 0) refers to the following camera: the position coordinate value of its center point is (10, 0, 200), and the angular position representing the posture is (180 degrees, 90 degrees, 0 degrees).
[0038] In the following, the position and / or posture is sometimes referred to as simply pose.
[0039] The display device may include one of a light emitting diode display (LED), a liquid crystal display (LCD), an organic light emitting diode display (OLED), a mini-OLED, and a projection screen. Hereinafter, the display device is sometimes referred to as a display.
[0040] The first mobile device may be a mobile robot (autonomous mobile robot AMR), which may be a robot that can be operated by a handle or may be a robot that can perform automatic driving, for example.
[0041] For example, the display is fixed on the mobile robot so that the display surface of the display is perpendicular to the ground and the edge is parallel to the ground.
[0042] As an example, the virtual background picture may be a picture of a computer graphics (CG) background. Those skilled in the art may also think of other examples of virtual background pictures, which are not described here. In the following, for simplicity, the virtual background is sometimes referred to as a CG background, and the virtual background picture is sometimes referred to as a CG background picture. Among them, the CG background picture is at least a part of the CG background.
[0043] Those skilled in the art may predefine at least one of the above-mentioned predetermined angles, sizes and positions based on experience or application scenarios.
[0044] The content production system that can be used to make movies and videos in the prior art may include a stage or performance area that is at least partially surrounded by one or more walls and / or ceilings, each of which may be covered with a display screen. One or more cameras may be placed in the performance area, and live actors may interact with physical props placed on the stage and virtual elements displayed on the display. Existing virtual production systems, such as studios, have a large performance area (e.g., 50-100 feet in diameter) and a large display (the display may be 20-40 feet high), and the system in the studio cannot move, let alone make the display move with the movement of the camera.
[0045] Compared with virtual production in a studio, the shooting device 100 according to the embodiment of the present disclosure has at least one of the following effects: it has a wider range of application scenarios (suitable for professionals such as professional film production teams and / or amateurs such as anchors, role-playing, etc., and can be used indoors and / or outdoors), simpler image post-processing (what you see is what you get), lower cost, and smaller footprint.
[0046] In a shooting system including the shooting device 100 according to an embodiment of the present disclosure, a movable display device provides a shooting background and can move along with the posture change of the shooting object and the shooting device 100, so that the virtual background picture has at least one of a predetermined angle, size, and position in the picture captured by the shooting device 100. Even if a large display device is not used, the shooting quality of the image and / or the video including the image can be guaranteed.
[0047] In addition, the autonomous movement of the virtual background screen used in photography activities, especially virtual production, can be achieved, and the entire shooting system can move with the target object so that the target object can more flexibly choose the shooting location. This movement is divided into two levels. On the physical level, the display device is fixed on the first mobile device. When the display device moves with the change of the posture of the shooting device 100, the virtual background screen moves with the change of the posture of the shooting device 100. On the content level, the virtual background screen can move with the posture of the shooting device 100, so that the display device providing the shooting background becomes a window to peek into the virtual world in the CG background, providing users with an immersive shooting experience.
[0048] As an example, the virtual background image has at least one of a predetermined angle, size, and position in the image captured by the shooting device 100, including: the proportion of the virtual background image in the image captured by the shooting device 100 is greater than a predetermined proportion threshold. Those skilled in the art can pre-set the predetermined proportion threshold based on experience or application scenarios. For example, when the camera posture changes, the first mobile device moves so that the camera can capture the image displayed by the display device to the greatest extent.
[0049] As an example, the proportion of the virtual background image in the image captured by the shooting device 100 is greater than a predetermined proportion threshold value, including: the center line of the field of view of the shooting device 100 forms a predetermined angle with the display surface of the display device and passes through the longitudinal center line of the display surface, wherein the longitudinal center line divides the display surface into equal parts in the vertical direction, in particular, the predetermined angle is 90 degrees, and / or the distance between the display device and the shooting device 100 is less than a predetermined distance threshold, so that the degree of overlap between the left and right ends of the display surface and the left and right ends of the image captured by the shooting device 100 is greater than a predetermined overlap threshold.
[0050] For example, the above "maximum degree" is described from two aspects: in terms of angle, it means that the center line of the camera's field of view is almost perpendicular to the display surface and almost passes through the longitudinal center line of the display surface (this center line divides the display surface into equal parts left and right, rather than up and down). In terms of distance, the display device is as close to the camera as possible, so that the left and right ends of the display surface overlap with the left and right ends of the picture taken by the camera as much as possible (that is, the picture is filled by the display surface); taking into account that the display device may be small in size or the camera shooting angle is wide, the user is allowed to adjust the angle and distance separately. Taking into account different shooting requirements, the user is allowed to limit the movement mode of the first mobile device through the user interface (UI). For example, the first mobile device is only allowed to rotate in place.
[0051] As an example, the shooting processing unit 102 may be configured to send the obtained relative position information between the shooting device 100 and the first mobile device to the processing device, so that the processing device can construct a virtual shooting environment in a three-dimensional space in real time based on the relative position information, wherein the virtual shooting environment includes a virtual background that cannot be moved and a virtual shooting device (also referred to as a virtual camera) that can move corresponding to the shooting device 100. For example, the virtual shooting device can reflect the position of the shooting device 100 in the three-dimensional virtual shooting environment.
[0052] As an example, at least a portion of the processing device is set in at least one of the photographing device 100, the display device, the first mobile device, and a server associated with the photographing device 100. Optionally, the processing device may also be independent of the photographing system, for example, the processing device may be a cloud server. The processing device may be, for example, a CPU (central processing unit), a GPU (graphics processing unit), a BPU (branch processing unit), etc. The processing device can wirelessly receive and send information between various parts of the photographing system, and can perform operations involved in the photographing system.
[0053] For example, the relative position information can be calculated and / or measured based on the following information: position information of the display device and / or the first mobile device acquired by at least one of the RGB sensor, the depth sensor, and the ToF sensor of the camera. The relative position information is sent to the processing device; the processing device models (hereinafter, also referred to as mapping) in real time based on this information to obtain a three-dimensional virtual shooting environment.
[0054] The virtual shooting environment includes at least a CG background and a virtual camera. In the initialization stage of modeling, a CG background can be selected by the user, for example, through a user interface. The CG background here can be a three-dimensional environment or a two-dimensional image. If a three-dimensional environment is selected, the virtual camera is loaded in the virtual shooting environment according to the optimal shooting angle of the target background; if a two-dimensional image is selected as the CG background, it is processed into a surface according to the camera intrinsic parameters and distortion parameters during loading, and the virtual camera is loaded on the horizontal center plane of the surface, and the distance from the virtual camera to the center of the surface is equal to the distance from the actual camera to the display surface during initialization. The posture of the virtual camera can be determined by the posture of the actual camera. After the initialization is completed, the processing device can update the posture of the virtual camera and the CG background screen displayed on the display device in real time according to the posture information of the camera and the display device.
[0055] As an example, the relative position information between the virtual shooting device corresponding to the shooting device 100 and the first mobile device can be used by the processing device to construct a virtual shooting environment in a three-dimensional space in real time, wherein the virtual shooting environment includes a virtual background that cannot be moved and a virtual shooting device that can be moved. That is, the virtual shooting environment can also be constructed based on the relative position information between the virtual shooting device and the first mobile device.
[0056] When the shooting system is turned on, it is necessary to perform environmental detection, that is, map building. Environmental detection will continue during the shooting process, that is, the map building will be updated. In addition, in order to prevent danger, it is also necessary to "scan" the working environment in some way and build a map. Map building or modeling can also be called environmental perception, environmental recognition, or environmental detection.
[0057] Environmental perception can be accomplished through a camera, which may involve at least one of a camera's RGB sensor, a depth sensor, a gyroscope, an acceleration sensor, and a ToF sensor.
[0058] Among them, the information collected by the RGB sensor, the depth sensor and the ToF sensor can be used for mapping; the information collected by the gyroscope and the acceleration sensor can constitute the position information of the camera, which is used to control the movement of the first mobile device, etc.
[0059] Generally, the sensing range of the depth sensor (eg, 10 m) is greater than the distance from the camera to the first mobile device (eg, 3 m).
[0060] When the shooting system is initialized, the first mobile device is not allowed to move, but the camera is used to shoot a circle. Since the first mobile device blocks a part of the field of view, the movement control of the first mobile device is enabled to complete the environmental information (including the environmental information corresponding to the blocked part).
[0061] As an example, the shooting processing unit 102 can be configured to send the obtained RGB information and / or depth information about the photographed object to the processing device, so that the processing device can obtain a three-dimensional model reflecting the position and / or outline of the photographed object in the virtual shooting environment based on the RGB information and / or depth information. When the subject appears in the camera's field of view, the virtual shooting environment also includes a three-dimensional model of the subject's position and / or outline information (for example, in the form of a three-dimensional point cloud, RGB map, etc.). The three-dimensional model of the subject's position and / or outline information is part of the above-mentioned environmental information. The three-dimensional model of the subject's position and / or outline information can prevent the equipment in the shooting system from colliding with the photographed object.
[0062] The situation that does not involve the robot arm holding the camera and does not involve the preset camera movement can be called the basic form of the shooting system. In the basic form, the camera can be held by a user (e.g., a photographer), and the user can decide how to move the camera.
[0063] An example of environment recognition in a basic form of a camera system is described in further detail below.
[0064] (1) System initialization: When the shooting system is initialized, the user starts the camera, the first mobile device, the display device, and the processing device; each device is initialized. When the display device is initialized, the user can select the CG background used for this shooting through the user interface (for example, the CG background can be selected from the resource library).
[0065] (2) establishing a connection between the camera, the first mobile device and the processing device to perform environment recognition;
[0066] (3) 3D environment recognition: The camera collects the position information of the display device and / or the first mobile device and sends it to the processing device. The processing device constructs a 3D environment virtual scene (virtual shooting environment), which includes, for example, a virtual camera (virtual shooting device) corresponding to the position of the real camera (shooting device 100) and a perspective for creating a CG background according to the position of the display device. As an example, 3D environment recognition can be performed through the following 5 steps:
[0067] Step 3-1: Get started.
[0068] Step 3-2: The camera obtains the position information of the display device and / or the first mobile device and the position information of the camera itself, and sends it to the processing device (for example, via WiFi, etc.).
[0069] Step 3-3: The processing device determines whether the currently acquired information meets the predetermined condition. If not, step 3-4 is executed. If the predetermined condition is met, step 3-5 is executed. Among them, those skilled in the art can pre-set the predetermined condition based on experience or application scenarios. For example, the predetermined condition can be to keep the display surface in the picture captured by the camera as large as possible while maintaining the distance from the center of the camera to the center of the display surface unchanged.
[0070] Step 3-4: The user holding the camera changes the position of the camera, and at the same time, the first mobile device moves with reference to the relative position information. Steps 3-2 and 3-3 are executed in a loop.
[0071] Step 3-5: End.
[0072] It should be noted that during the system initialization process, in order to avoid missing environmental information due to occlusion, the camera needs to be moved (such as horizontal rotation, etc.) for multi-angle sampling. In this process, the first mobile device moves with the change of the camera's posture, so that the camera can capture the picture displayed on the display surface to the greatest extent.
[0073] Figure 3 FIG. 4 is a flowchart showing an example of mapping and obtaining a three-dimensional model of a subject during a photographing process according to an embodiment of the present disclosure.
[0074] In S31, the position and posture of the camera changes.
[0075] In S32 , the camera sends relative position information, RGB information and / or depth information to the processing device.
[0076] In S33, the processing device creates a map based on the relative position information.
[0077] In S34, the processing device obtains a three-dimensional model of the object according to the RGB information and / or the depth information.
[0078] In S35, the map is constructed.
[0079] As an example, at least a portion of the virtual background observed by the virtual shooting device is displayed on the display device as a virtual background picture, and the virtual background picture is dynamically updated based on the position and / or posture of the virtual shooting device, and thus is dynamically updated based on the position and / or posture of the shooting device 100.
[0080] In the virtual shooting environment constructed by the processing device, the virtual camera can change its position in real time according to the position change of the real camera, while the CG background remains stationary, thus achieving the effect of the virtual camera moving while observing different areas of the CG background. As a result, the scene observed by the virtual camera is rendered on the display surface in real time, thereby giving the CG background ductility and enhancing the immersive feeling of the shooting activity.
[0081] Figure 4 1 is an example flowchart showing screen control of a display device when the posture of a photographing device (eg, a camera) changes according to an embodiment of the present disclosure.
[0082] In S41, the position and posture of the camera changes.
[0083] In S42, the camera sends camera pose information to the processing device.
[0084] In S43, the processing device updates the pose of the virtual camera according to the camera pose information.
[0085] In S44, the virtual background image captured by the virtual camera changes.
[0086] In S45 , the virtual background image is sent to the display device.
[0087] For example, when the camera moves, the virtual camera in the 3D rendering software of the PC moves accordingly. The display device is equivalent to broadcasting the scene captured by the virtual camera (virtual background picture).
[0088] The shooting system of the present application can realize dynamic shooting. That is, when the position of the camera changes, the first mobile device moves along with it, for example, so that the camera can capture the picture displayed on the display surface to the greatest extent.
[0089] The implementation example of dynamic shooting may include the following 6 steps:
[0090] 1) Start.
[0091] 2) Initialization of the 3D virtual shooting environment. Obtain the shooting background specified by the user, create the corresponding CG background in the virtual environment, and initialize the virtual camera pose according to the aforementioned relative position information. After the initialization is completed, the position of the CG background in the virtual environment remains unchanged.
[0092] 3) After initialization is completed, the position of the virtual camera is updated in real time according to the position of the real camera.
[0093] 4) The processing device renders the background (virtual background image) presented on the display surface in real time according to the CG background observed from the virtual camera.
[0094] 5) The first mobile device moves in real time according to the changes of the virtual camera, so that, for example, the camera can capture the picture displayed on the display surface to the greatest extent.
[0095] 6) End.
[0096] In the present application, the user is allowed to choose to manually control the first mobile device, or to let the first mobile device be automatically controlled according to the dynamic shooting mode. In the manual control mode, the joystick can be used to control the first mobile device.
[0097] The significance of the manual control mode is that 1) before shooting, it is convenient for the user to move the first mobile device to the shooting location; 2) during the shooting process, the camera posture may change suddenly (for example, the photographer confirms the effect of the previous photo, etc.), and the manual control mode can avoid the danger caused by sudden movement of the camera.
[0098] As an example, in the automatic control mode, the position to which the first mobile device is to be moved may be calculated based on the position and / or posture of the photographing apparatus 100 .
[0099] Figure 5 Detailed description is a flowchart showing an example of controlling movement of a first mobile device when the posture of a photographing device (eg, a camera) changes according to an embodiment of the present disclosure.
[0100] In S51, the position and posture of the camera changes.
[0101] In S52, the camera sends camera pose information to the processing device.
[0102] In S53, the processing device calculates the position to which the first mobile device is to move (should be at the position) according to the camera posture information.
[0103] In S54, the processing device performs motion planning for the first mobile device and sends an instruction to the first mobile device. For example, the processing device performs motion planning based on the above environmental information.
[0104] For example, the camera moves, so that the expected position of the first mobile device changes. The position of the first mobile device can be controlled based on the expected position and the actual position of the first mobile device.
[0105] The first mobile device moves according to the motion instruction, its position changes, and its position information is sent to the processing device in real time. The position information of the first mobile device can come from the sensor of the first mobile device itself, and / or the position information of the first mobile device can also be obtained from the image information of the camera by an image recognition method (for example, the camera sends the image information it obtains to the processing device, and the processing device obtains the position information of the first mobile device based on the image recognition algorithm).
[0106] As an example, the camera 100 is held by a robotic arm fixed at the bottom, the camera 100 is fixed to the end part of the robotic arm, and the fixing part between the camera 100 and the end part of the robotic arm makes the shooting center line of the camera 100 parallel to the center line of the end part. For example, the gripper end of the robotic arm holds the camera 100 by carrying a gimbal. The robotic arm used to hold the camera reflects the scalability of the shooting system. The robotic arm is used to adjust the position and posture of the camera to match the movement of the target object. The robotic arm can realize automatic holding and movement of the camera during shooting.
[0107] Figure 6 FIG. 1 is a schematic diagram showing a shooting center line of the shooting device 100 according to an embodiment of the present disclosure. Figure 6 As shown, the photographing device 100 is held by a robot arm. Figure 7 FIG. 1 is a schematic diagram showing that the shooting center line of the shooting device 100 according to an embodiment of the present disclosure is parallel to the center line of the end component. Figure 7 As shown, the shooting center line is parallel to the center line of the end component.
[0108] As mentioned above, in the basic form, it is up to the user, such as a photographer, to decide how to move the camera. In the scenario where a robotic arm holds the camera, the user needs to tell the shooting system how the robotic arm should move (move the camera). For example, the user can specify the movement of the virtual camera through a user interface. This UI can be, for example, commonly used rendering software. In other words, the position of the virtual camera may not be driven by the position information of the real camera, but may be controlled by the user in the UI. This provides the option of manually adjusting the CG background.
[0109] The camera is fixed to the end member of the robot arm, so once the camera motion is specified, the motion target of the end member of the robot arm is also determined (ie, the time series of the target pose of the robot arm is determined).
[0110] Based on this motion target, the robotic arm can be controlled through a motion planning algorithm.
[0111] Figure 8 is an example flow chart illustrating the control of the motion of a robotic arm according to an embodiment of the present disclosure.
[0112] In S81 , the user specifies the movement of the virtual camera in the user interface, such as trajectory, speed, acceleration, etc.
[0113] In S82, the processing device calculates the motion target of the robot arm.
[0114] In S83, the user issues a motion instruction through the user interface.
[0115] In S84, the robot arm starts to move, and the position of the camera changes accordingly.
[0116] As the camera pose changes, according to the reference Figure 5 The described method controls the movement of a first mobile device equipped with a display device.
[0117] As an example, the shooting processing unit 102 can be configured to send the obtained image of the shooting object to the processing device, so that the processing device can identify the position and / or posture of the shooting object based on the image to determine the shooting position and / or moving route of the virtual shooting device 100, thereby determining the shooting position and / or moving route of the shooting device 100.
[0118] The shooting system of the present application can also use the visual information of the subject's posture obtained by the camera to infer the best shooting position through an algorithm, and then automatically move the camera to the best position for shooting through a robotic arm. For example, the movement line or position of the virtual camera may not be set by the user, but the processing device may infer the optimal shooting position or movement line based on the position, posture and other information of the subject observed by the camera. For example, the camera can communicate with the processing device, send the image captured by the camera to the processing device in real time, and receive shooting instructions for operating the shutter, and receive parameter setting instructions for adjusting the focal length, etc. In other words, it is possible to infer the best shooting angle and parameters based on the posture of the subject, and perform automatic shooting. The implementation of automatic shooting may, for example, include the following steps:
[0119] 1) Start automatic shooting mode.
[0120] 2) The image of the person being photographed captured by the camera is sent to the processing device in real time.
[0121] 3) The processing device recognizes the posture and position of the subject based on the image, for example, by using a posture recognition algorithm.
[0122] 4) Determine the best shooting composition and camera position based on a preset pose-composition mapping set (by the software developer, user, or other developer).
[0123] 5) Taking the best shooting position as the target position, command the virtual camera to move.
[0124] 6) According to the aforementioned dynamic shooting function, the first mobile device moves, and the content rendered by the display device (virtual background image) changes accordingly.
[0125] 7) After the camera and the first mobile device arrive at the target location, the processing device sends instructions, for example, through the communication module to perform pre-shooting preparation operations such as parameter adjustment and focusing.
[0126] 8) The system sends a prompt to the person being photographed (such as a shooting countdown, etc.). The prompt here can be a camera indicator light, a camera buzzer, etc.
[0127] 9) The processing device sends an instruction, for example, through its communication module, to press the shutter.
[0128] 10) After the shooting is completed, the shooting system will prompt the photographer.
[0129] 11) End.
[0130] In the above-mentioned automatic shooting scenario, there may be a situation where it is necessary to issue instructions and guidance to the subject. A voice interaction module can be introduced, which, for example, includes a speaker, a microphone, and a communication unit. This voice interaction module can be placed on one of the display device, the shooting device 100, and the first mobile device. In the automatic shooting scenario, the processing device gives voice prompts to the subject (countdown before shooting, shooting completion, etc.). In addition, the processing device makes a judgment based on the posture position of the subject, the position of the display device, and the best composition; if the camera has reached the best position, a voice prompt can be given to the subject whose position deviates from the position required by the best composition. At the same time, the subject can also give instructions to the system through voice commands, such as shooting, fixing the display surface, fixing the camera, etc. The voice interaction module further reflects the scalability of the shooting system.
[0131] As an example, the bottom end of the robot arm is fixed to the second mobile device, and the robot arm and the second mobile device are coupled (linked) via a lifting mechanism for lifting the robot arm, and the robot arm is coupled to the lifting mechanism via a fixing member.
[0132] In some cases, the camera may be required to move to a position far from the initial position, in which case a second mobile device is required to move the camera to the target position while avoiding tipping over.
[0133] The second mobile device may be a mobile robot (autonomous mobile robot AMR), which may be a robot that can be operated by a handle or may be a robot that can perform automatic driving, for example.
[0134] Fig. 9 FIG. 2 shows a schematic diagram of a mechanical arm fixed to a second mobile device according to an embodiment of the present disclosure. Fig. 9 As shown, the mechanical arm is coupled to the second mobile device via a lifting mechanism.
[0135] As an example, the motion of the robot and / or the second mobile device is calculated based on the motion of the pre-specified virtual photographing device, and the motion of the photographing device 100 is determined based on the calculated motion of the robot and / or the second mobile device.
[0136] Fig.10 is an example flow chart illustrating controlling the movement of a robotic arm and a second mobile device according to an embodiment of the present disclosure.
[0137] In S1001 , the user specifies the movement of the virtual camera in the user interface, such as trajectory, speed, acceleration, etc.
[0138] In S1002 , the processing device calculates the motion targets of the robotic arm and the second mobile device.
[0139] In S1003 , the user issues a motion instruction through the user interface.
[0140] In S1004, the robotic arm and the second mobile device start to move, and the camera posture changes accordingly.
[0141] As the camera pose changes, according to the reference Figure 5 The described method controls the movement of a first mobile device equipped with a display device.
[0142] The present disclosure also provides a display device. Fig.11 FIG. 1 shows a functional module block diagram of a display device 1100 according to an embodiment of the present disclosure. Fig.11 As shown, the display device 1100 includes: a display control unit 1101, which can be configured to control the display device 1100, and a display processing unit 1102, which can be configured to enable the display unit included in the display device 1100 to display a virtual background picture so that a shooting device that can move around the subject can capture at least one image including the subject and the virtual background picture.
[0143] The display control unit 1101 and the display processing unit 1102 may be implemented by one or more processing circuits, which may be implemented as a chip or a processor, for example. Fig.11 The various functional units shown in the figure are merely logical modules divided according to the specific functions they implement, and are not intended to limit the specific implementation method.
[0144] The display device 1100 may be implemented at a chip level or may be implemented at a device level.
[0145] The photographed object is located between the photographing device and the display device 1100, and the display device 1100 is set on the first mobile device so that it can be moved by the first mobile device, so that the virtual background picture has at least one of a predetermined angle, size and position in the picture captured by the photographing device, and the virtual background picture is dynamically updated based on the position and / or posture of the photographing device.
[0146] Fig.121 is a schematic diagram showing an example of a display device 1100 according to an embodiment of the present disclosure. Fig.12 As shown, the display device 1100 is set on the first mobile device.
[0147] The display device 1100 may be the display device in the embodiment of the photographing device 100 , the photographing device in the embodiment of the display device 1100 may be the photographing device 100 , and the first mobile device in the embodiment of the display device 1100 may be the first mobile device in the embodiment of the photographing device 100 .
[0148] The display device 1100 may include one of a light emitting diode display (LED), a liquid crystal display (LCD), an organic light emitting diode display (OLED), a mini-OLED, and a projection screen. Hereinafter, the display device 1100 is sometimes referred to as a display.
[0149] As an example, the first mobile device may be a mobile robot.
[0150] For details about the photographing device, the first mobile device, the position and / or posture of the photographing device, and the virtual background image, please refer to the relevant description in the embodiment of the photographing device 100, which will not be repeated here.
[0151] Those skilled in the art may predefine at least one of the above-mentioned predetermined angles, sizes and positions based on experience or application scenarios.
[0152] Compared with virtual production in a studio, the display device 1100 according to the embodiment of the present disclosure has at least one of the following effects: a wider range of application scenarios (suitable for professionals such as professional film production teams and / or amateurs such as anchors, role-playing, etc., and can be used indoors and / or outdoors), simpler image post-processing (what you see is what you get), lower cost, and smaller footprint.
[0153] According to the embodiment of the present disclosure, the movable display device 1100 can move along with the posture change of the photographed object and the photographing device, so that the virtual background image has at least one of a predetermined angle, size and position in the image captured by the photographing device. Even if a large display device is not used, the shooting quality of the image and / or the video including the image can be guaranteed.
[0154] In addition, it can realize the autonomous movement of virtual background images used in photography activities, especially virtual production, to provide users with an immersive shooting experience.
[0155] As an example, the virtual background picture having at least one of a predetermined angle, size, and position in the picture captured by the shooting device may include: the proportion of the virtual background picture in the picture captured by the shooting device is greater than a predetermined proportion threshold. Those skilled in the art may pre-set the predetermined proportion threshold based on experience or application scenarios.
[0156] As an example, the proportion of the virtual background picture in the picture captured by the shooting device being greater than a predetermined proportion threshold may include: the center line of the field of view of the shooting device forms a predetermined angle with the display surface of the display device 1100 and passes through the longitudinal center line of the display surface, wherein the longitudinal center line divides the display surface into equal parts in the vertical direction, in particular, the predetermined angle is 90 degrees, and / or the distance between the display device 1100 and the shooting device is less than a predetermined distance threshold, so that the degree of overlap between the range between the left and right ends of the display surface and the range between the left and right ends of the picture captured by the shooting device is greater than a predetermined overlap threshold.
[0157] As an example, the relative position information between the photographing device and the first mobile device is used to construct a virtual photographing environment in a three-dimensional space in real time, wherein the virtual photographing environment includes an immovable virtual background and a movable virtual photographing device corresponding to the photographing device.
[0158] As an example, relative position information between a virtual photographing device corresponding to the photographing device and the first mobile device is used to construct a virtual photographing environment in a three-dimensional space in real time, wherein the virtual photographing environment includes an immovable virtual background and a movable virtual photographing device.
[0159] For the virtual shooting environment, please refer to the relevant description in the embodiment of the shooting device 100, which will not be repeated here.
[0160] As an example, at least a portion of the virtual background observed by the virtual shooting device is displayed on the display device 1100 as a virtual background picture, and the virtual background picture is dynamically updated based on the position and / or posture of the virtual shooting device, thereby being dynamically updated based on the position and / or posture of the shooting device.
[0161] For details about the virtual background image being dynamically updated, please refer to the embodiment of the shooting device 100. Figure 4 The relevant descriptions will not be repeated here.
[0162] As an example, the position to which the first mobile device is to be moved is calculated based on the position and / or posture of the camera.
[0163] For a description of the position to which the first mobile device is to be moved, see the embodiment of the photographing device 100. Figure 5 The relevant descriptions will not be repeated here.
[0164] As an example, the display device 1100 is disposed on the first mobile device via a connecting member, and the weight of the display device 1100 is borne by the first mobile device and a spherical wheel located at the bottom of the connecting member.
[0165] Fig.13 Schematic diagram showing a connection component according to an embodiment of the present disclosure. Fig.13 As shown, the display device 1100 is arranged on the first mobile device via a connection component.
[0166] Fig.14 Schematic diagram of a spherical wheel according to an embodiment of the present disclosure. Fig.14 As shown, the weight of the display device 1100 is shared by the first moving device and the spherical wheels located at the bottom of the vertical beam, rather than being fully supported by the first moving device.
[0167] As an example, the spherical wheel is a driven wheel that can move in various directions. That is, the spherical wheel is used as the driven wheel, so it can move freely in various directions.
[0168] The present disclosure also provides a shooting system.
[0169] According to an embodiment of the present disclosure, a shooting system includes: a display device, which displays a virtual background picture, and a shooting device, which can move around a shooting object to capture at least one image including the shooting object and the virtual background picture, wherein the shooting object is located between the shooting device and the display device, and the display device is set on a first mobile device and can be moved by the first mobile device so that the virtual background picture has at least one of a predetermined angle, size and position in the picture captured by the shooting device, and the virtual background picture is dynamically updated based on the position and / or posture of the shooting device.
[0170] In the shooting system, the shooting device may be the shooting device 100 , the display device may be the display device 1100 or the display device in the embodiment of the shooting device 100 , and the first mobile device may be the first mobile device in the embodiment of the shooting device 100 .
[0171] The photographing device is, for example, a camera, which is, for example, a digital camera, which has at least one of the functions of an RGB sensor, a depth sensor, a ToF (time of flight) sensor, and a 6-axis acceleration sensor (i.e., an acceleration sensor and a gyroscope), and has a communication function that can send information to the outside in a wireless form and can receive information from the outside. The photographing device is, for example, other devices that can realize the above functions, which will not be repeated here.
[0172] The display device may include one of a light emitting diode display (LED), a liquid crystal display (LCD), an organic light emitting diode display (OLED), a mini-OLED, and a projection screen.
[0173] As an example, the first mobile device may be a mobile robot.
[0174] For details about the photographing device, the first mobile device, the position and / or posture of the photographing device, and the virtual background image, please refer to the relevant description in the embodiment of the photographing device 100, which will not be repeated here.
[0175] Those skilled in the art may predefine at least one of the above-mentioned predetermined angles, sizes and positions based on experience or application scenarios.
[0176] Compared with virtual production in a studio, the shooting system according to the embodiment of the present disclosure has at least one of the following effects: it has a wider range of application scenarios (suitable for professionals such as professional film production teams and / or amateurs such as anchors, role-playing, etc., and can be used indoors and / or outdoors), simpler image post-processing (what you see is what you get), lower cost, and smaller footprint.
[0177] In the shooting system according to the embodiment of the present disclosure, the movable display device can move along with the posture change of the shooting object and the shooting device, so that the virtual background picture has at least one of a predetermined angle, size and position in the picture captured by the shooting device. Even if a large display device is not used, the shooting quality of the image and / or the video including the image can be guaranteed.
[0178] In addition, the autonomous movement of virtual background images used in photography activities, especially virtual production, can be achieved, and the entire shooting system can move with the target object, so that the target object can choose the shooting location more flexibly, providing users with an immersive shooting experience.
[0179] As an example, the virtual background picture having at least one of a predetermined angle, size and position in the picture captured by the shooting device may include: a proportion of the virtual background picture in the picture captured by the shooting device is greater than a predetermined proportion threshold.
[0180] As an example, the proportion of the virtual background picture in the picture captured by the shooting device being greater than a predetermined proportion threshold may include: the center line of the field of view of the shooting device forms a predetermined angle with the display surface of the display device and passes through the longitudinal center line of the display surface, wherein the longitudinal center line divides the display surface into equal parts in the vertical direction, in particular, the predetermined angle is 90 degrees, and / or the distance between the display device and the shooting device is less than a predetermined distance threshold, so that the degree of overlap between the range between the left and right ends of the display surface and the range between the left and right ends of the picture captured by the shooting device is greater than a predetermined overlap threshold.
[0181] As an example, the shooting device sends the obtained relative position information between the shooting device and the first mobile device to the processing device, so that the processing device constructs a virtual shooting environment in three-dimensional space in real time based on the relative position information, wherein the virtual shooting environment includes an immovable virtual background and a movable virtual shooting device corresponding to the shooting device.
[0182] As an example, the relative position information between the virtual shooting device corresponding to the shooting device and the first mobile device is used by the processing device to construct a virtual shooting environment in three-dimensional space in real time, wherein the virtual shooting environment includes an immovable virtual background and a movable virtual shooting device.
[0183] For the virtual shooting environment, please refer to the relevant description in the embodiment of the shooting device 100, which will not be repeated here.
[0184] As an example, at least a part of the processing device is set in at least one of the shooting device, the display device, the first mobile device, and a server associated with the shooting system. Optionally, the processing device can also be independent of the shooting system, for example, the processing device can be a cloud server.
[0185] As an example, at least a portion of the virtual background observed by the virtual shooting device is displayed on the display device as a virtual background picture, and the virtual background picture is dynamically updated based on the position and / or posture of the virtual shooting device, thereby being dynamically updated based on the position and / or posture of the shooting device.
[0186] For details about the virtual background image being dynamically updated, please refer to the embodiment of the shooting device 100. Figure 4 The relevant descriptions will not be repeated here.
[0187] As an example, the photographing device sends the obtained RGB information and / or depth information about the photographed object to the processing device, so that the processing device obtains a three-dimensional model reflecting the position and / or outline of the photographed object in the virtual photographing environment based on the RGB information and / or depth information.
[0188] As an example, the photographing device is held by a robotic arm fixed at the bottom, the photographing device is fixed to the end part of the robotic arm, and the fixing part between the photographing device and the end part of the robotic arm makes the photographing center line of the photographing device parallel to the center line of the end part.
[0189] As an example, the bottom end of the robot arm is fixed to the second mobile device, and the robot arm and the second mobile device are coupled via a lifting mechanism for lifting the robot arm, and the robot arm is coupled to the lifting mechanism via a fixing member.
[0190] As an example, the second mobile device may be a mobile robot.
[0191] As an example, the movement to be performed by the robot arm and / or the second mobile device is calculated based on the movement of the pre-specified virtual camera, and the movement of the camera is determined based on the calculated movement to be performed by the robot arm and / or the second mobile device.
[0192] For an example process of controlling the motion of the mechanical arm and the second mobile device, see the embodiment of the shooting device 100 in combination with Fig.10 The relevant descriptions will not be repeated here.
[0193] As an example, the photographing device sends the obtained image of the photographing object to the processing device, so that the processing device can identify the position and / or posture of the photographing object based on the image to determine the photographing position and / or moving route of the virtual photographing device, thereby determining the photographing position and / or moving route of the photographing device.
[0194] As an example, the position to which the first mobile device is to be moved is calculated based on the position and / or posture of the camera.
[0195] For a description of the position to which the first mobile device is to be moved, see the embodiment of the photographing device 100. Figure 5 The relevant descriptions will not be repeated here.
[0196] As an example, the display device is arranged on the first mobile device via a connecting component, and the weight of the display device is borne by the first mobile device and a spherical wheel located at the bottom of the connecting component.
[0197] As an example, a spherical wheel is a driven wheel that can move in all directions.
[0198] As an example, the shooting system also includes an anti-misoperation component, which prevents the first mobile device from moving rapidly at a speed exceeding a predetermined speed threshold when it detects that the change in the position and / or posture of the shooting device exceeds a predetermined change threshold and / or the rate of change of the position and / or posture of the shooting device exceeds a predetermined change rate threshold, and the center line of the field of view of the shooting device exceeds the range of the display surface of the display device. For example, in a dynamic shooting mode, the anti-misoperation component can monitor the change and / or rate of change of the camera posture, and the relationship between the camera viewing angle and the display surface based on the camera posture information and the display device position information. For example, when the processing device detects that the camera posture changes suddenly, if the center line of the camera's field of view exceeds the range of the display surface, steps 4) and 5) of the six steps included in the implementation example of dynamic shooting above are terminated, thereby preventing the first mobile device from moving suddenly and rapidly, thereby improving safety and improving shooting quality.
[0199] As an example, the shooting system further includes a lighting component, which provides a fill light source for shooting performed by the shooting device, and sends its position information to the processing device, so as to reproduce the fill light source in the virtual shooting environment of the three-dimensional space constructed by the processing device. For example, the lighting component has a communication unit and a position information sensing unit, and the lighting component can send its own position information to the processing device, so as to reproduce the light source and the light path in the aforementioned three-dimensional virtual shooting.
[0200] As an example, the photographing system further includes a shadow removing component for removing a shadow of the photographing object caused by the lighting component and appearing on the display surface of the display device.
[0201] Fig.15 An example of a shadow of a photographic subject appearing on a display surface of a display device due to lighting components is shown.
[0202] As an example, the shadow removal component estimates the corresponding area of the shadow of the subject on the display surface based on the position information of the lighting component, the image information of the subject obtained by the shooting device, and the relative position information between the shooting device and the display device, and performs shadow compensation processing on the corresponding area (for example, rendering the pixels at the shadow position), thereby eliminating the shadow on the captured virtual background picture. For example, the shadow removal component calculates based on the position information of the lighting component, the image information of the subject obtained by the camera, and the relative position information between the display and the camera in the system, and infers the area affected by the shadow of the subject on the display surface, and then performs shadow compensation processing on the area within the shadow when rendering the CG background, thereby eliminating the shadow on the captured background, that is, eliminating the shadow of the subject on the display surface caused by the lighting component, so that the shadow cannot be observed in the captured picture, thereby improving the shooting quality.
[0203] Anti-misoperation components, lighting components, and shadow elimination components further reflect the scalability of the shooting system.
[0204] As an example, the first mobile device calculates the reflective area where the lighting component reflects light on the shooting device due to the mirror reflection of the display surface during shooting, and the shooting device moves away from the reflective area based on the light source direction of the lighting component, the material of the display surface, and the ambient light. For example, it is necessary to avoid the lighting component that provides light source for shooting from reflecting light on the camera due to the mirror reflection of the display device during shooting, thereby affecting the shooting effect. After the position of the lighting component is determined, the reflective area can be calculated based on the light source direction of the lighting component, the material of the display surface, and the ambient light according to, for example, the Phong lighting model. The shooting device is made to move autonomously to avoid the reflective area to prevent the lighting component from affecting the shooting picture, that is, to avoid the reflective effect of the lighting component due to mirror reflection in the camera's field of view.
[0205] The basic principles of the present invention are described above in conjunction with specific embodiments. However, it should be pointed out that those skilled in the art can understand that all or any components of the apparatus and device of the present invention and the system can be implemented in any computing device (including a processor, a storage medium, etc.) or a network of computing devices in the form of hardware, firmware, software, or a combination thereof. This can be achieved by those skilled in the art using their basic circuit design knowledge or basic programming skills after reading the description of the present invention.
[0206] Furthermore, the present invention also provides a program product storing machine-readable instruction codes. When the instruction codes are read and executed by a machine, the functions of the above-mentioned apparatus, device and system according to the embodiments of the present invention can be executed.
[0207] Accordingly, the storage medium for carrying the program product storing the machine-readable instruction code is also included in the disclosure of the present invention. The storage medium includes but is not limited to a floppy disk, an optical disk, a magneto-optical disk, a memory card, a memory stick, and the like.
[0208] When the present invention is implemented by software or firmware, the data is transferred from a storage medium or a network to a computer (eg, Fig.16 The general-purpose computer 1600 shown in the figure is installed with the programs constituting the software, and when the various programs are installed, the computer can execute various functions, etc.
[0209] exist Fig.16 In the embodiment of the present invention, a central processing unit (CPU) 1601 performs various processes according to a program stored in a read-only memory (ROM) 1602 or a program loaded from a storage section 1608 to a random access memory (RAM) 1603. In the RAM 1603, data required when the CPU 1601 performs various processes and the like is also stored as needed. The CPU 1601, the ROM 1602, and the RAM 1603 are connected to each other via a bus 1604. An input / output interface 1605 is also connected to the bus 1604.
[0210] The following components are connected to the input / output interface 1605: an input section 1606 (including a keyboard, a mouse, etc.), an output section 1607 (including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.), a storage section 1608 (including a hard disk, etc.), a communication section 1609 (including a network interface card such as a LAN card, a modem, etc.). The communication section 1609 performs communication processing via a network such as the Internet. A drive 1610 may also be connected to the input / output interface 1605 as needed. A removable medium 1611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is installed on the drive 1610 as needed, so that a computer program read therefrom is installed in the storage section 1608 as needed.
[0211] In the case where the above-described series of processing is realized by software, a program constituting the software is installed from a network such as the Internet or a storage medium such as the removable medium 1611 .
[0212] It should be understood by those skilled in the art that such storage media is not limited to Fig.16 The removable medium 1611 shown has a program stored therein and is distributed separately from the device to provide the program to the user. Examples of the removable medium 1611 include magnetic disks (including floppy disks (registered trademark)), optical disks (including compact disk read-only memory (CD-ROM) and digital versatile disks (DVD)), magneto-optical disks (including minidiscs (MD) (registered trademark)), and semiconductor memories. Alternatively, the storage medium may be the ROM 1602, a hard disk included in the storage section 1608, or the like, in which the program is stored and distributed to the user together with the device containing them.
[0213] It should also be noted that in the apparatus, device and system of the present invention, each component can be decomposed and / or reassembled. These decompositions and / or reassemblies should be regarded as equivalent solutions of the present invention. In addition, the steps of performing the above series of processes can naturally be performed in chronological order according to the order of description, but do not necessarily need to be performed in chronological order. Some steps can be performed in parallel or independently of each other.
[0214] Finally, it should be noted that the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In addition, in the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, article or device that includes the elements.
[0215] Although the embodiments of the present invention are described in detail above in conjunction with the accompanying drawings, it should be understood that the above-described embodiments are only used to illustrate the present invention and do not constitute a limitation of the present invention. For those skilled in the art, various modifications and changes can be made to the above-mentioned embodiments without departing from the essence and scope of the present invention. Therefore, the scope of the present invention is limited only by the attached claims and their equivalent meanings.
[0216] The present technology can also be implemented as follows.
[0217] Note 1. A photographing device, comprising:
[0218] a processing circuit configured to control so that the shooting unit included in the shooting device moves around the shooting object to capture at least one image including the shooting object and a virtual background picture displayed on the display device,
[0219] The photographed object is located between the photographing device and the display device.
[0220] The display device is arranged on the first mobile device and can be moved by the first mobile device so that the virtual background image has at least one of a predetermined angle, size and position in the image captured by the shooting device, and
[0221] The virtual background image is dynamically updated based on the position and / or posture of the camera.
[0222] Appendix 2. The photographing device according to Appendix 1, wherein:
[0223] The virtual background picture has at least one of a predetermined angle, size and position in the picture captured by the shooting device, including: the proportion of the virtual background picture in the picture captured by the shooting device is greater than a predetermined proportion threshold.
[0224] Supplement 3. The photographing device according to Supplement 2, wherein:
[0225] The proportion of the virtual background image in the image captured by the shooting device is greater than a predetermined proportion threshold, including:
[0226] The center line of the field of view of the camera device forms a predetermined angle with the display surface of the display device and passes through the longitudinal center line of the display surface, wherein the longitudinal center line divides the display surface into equal parts in the vertical direction, in particular, the predetermined angle is 90 degrees, and / or
[0227] The distance between the display device and the shooting device is less than a predetermined distance threshold, so that the degree of overlap between the left and right ends of the display surface and the left and right ends of the picture captured by the shooting device is greater than a predetermined overlap threshold.
[0228] Note 4. The photographing device according to any one of Notes 1 to 3, wherein:
[0229] The processing circuit is configured to send the obtained relative position information between the shooting device and the first mobile device to the processing device, so that the processing device can construct a virtual shooting environment in a three-dimensional space in real time based on the relative position information, wherein the virtual shooting environment includes an immovable virtual background and a movable virtual shooting device corresponding to the shooting device.
[0230] Note 5. A display device according to any one of Notes 1 to 3, wherein relative position information between a virtual shooting device corresponding to the shooting device and the first mobile device is used by a processing device to construct a virtual shooting environment in a three-dimensional space in real time, wherein the virtual shooting environment includes an immovable virtual background and the movable virtual shooting device.
[0231] Supplement 6. The photographing device according to Supplement 4 or 5, wherein:
[0232] At least a portion of the virtual background observed by the virtual shooting device is displayed on the display device as the virtual background picture, and
[0233] The virtual background picture is dynamically updated based on the position and / or posture of the virtual shooting device, thereby being dynamically updated based on the position and / or posture of the shooting device.
[0234] Note 7. The photographing device according to any one of Notes 4 to 6, wherein:
[0235] The processing circuit is configured to send the obtained RGB information and / or depth information about the photographed object to the processing device, so that the processing device obtains a three-dimensional model reflecting the position and / or outline of the photographed object in the virtual photographing environment based on the RGB information and / or the depth information.
[0236] Note 8. The photographing device according to any one of Notes 4 to 7, wherein:
[0237] The shooting device is held by a mechanical arm fixed at the bottom.
[0238] The photographing device is fixed to the end part of the robot arm, and
[0239] The fixing component between the photographing device and the end component of the robot arm makes the photographing center line of the photographing device parallel to the center line of the end component.
[0240] Supplement 9. The photographing device according to Supplement 8, wherein:
[0241] The bottom end of the mechanical arm is fixed to the second mobile device, and
[0242] The mechanical arm is coupled to the second moving device via a lifting mechanism for lifting the mechanical arm, and the mechanical arm is coupled to the lifting mechanism via a fixing component.
[0243] Note 10. The photographing device according to Note 8 or 9, wherein:
[0244] The movement of the robot arm and / or the second mobile device is calculated based on the pre-specified movement of the virtual camera, and the movement of the camera is determined based on the calculated movement of the robot arm and / or the second mobile device.
[0245] Supplement 11. The photographing device according to Supplement 8 or 9, wherein:
[0246] The processing circuit is configured to send the obtained image of the photographed object to the processing device, so that the processing device can identify the position and / or posture of the photographed object based on the image to determine the shooting position and / or moving route of the virtual shooting device, thereby determining the shooting position and / or moving route of the shooting device.
[0247] Note 12. The photographing device according to any one of Notes 4 to 11, wherein:
[0248] At least a part of the processing device is arranged in at least one of the photographing device, the display device, the first mobile device, and a server associated with the photographing device.
[0249] Note 13. A display device comprising:
[0250] a processing circuit configured to control so that a display unit included in the display device displays a virtual background picture, so that a shooting device capable of moving around a shooting object can capture at least one image including the shooting object and the virtual background picture,
[0251] The photographed object is located between the photographing device and the display device.
[0252] The display device is arranged on the first mobile device so as to be moved by the first mobile device, so that the virtual background image has at least one of a predetermined angle, size and position in the image captured by the shooting device, and
[0253] The virtual background image is dynamically updated based on the position and / or posture of the camera.
[0254] Supplement 14. The display device according to Supplement 13, wherein:
[0255] The virtual background picture has at least one of a predetermined angle, size and position in the picture captured by the shooting device, including: the proportion of the virtual background picture in the picture captured by the shooting device is greater than a predetermined proportion threshold.
[0256] Note 15. The display device according to Note 14, wherein:
[0257] The proportion of the virtual background image in the image captured by the shooting device is greater than a predetermined proportion threshold, including:
[0258] The center line of the field of view of the camera device forms a predetermined angle with the display surface of the display device and passes through the longitudinal center line of the display surface, wherein the longitudinal center line divides the display surface into equal parts in the vertical direction, in particular, the predetermined angle is 90 degrees, and / or
[0259] The distance between the display device and the shooting device is less than a predetermined distance threshold, so that the degree of overlap between the left and right ends of the display surface and the left and right ends of the picture captured by the shooting device is greater than a predetermined overlap threshold.
[0260] Note 16. A display device according to any one of Notes 13 to 15, wherein relative position information between the shooting device and the first mobile device is used to construct a virtual shooting environment in a three-dimensional space in real time, wherein the virtual shooting environment includes an immovable virtual background and a movable virtual shooting device corresponding to the shooting device.
[0261] Note 17. A display device according to any one of Notes 13 to 15, wherein relative position information between a virtual shooting device corresponding to the shooting device and the first mobile device is used to construct a virtual shooting environment in a three-dimensional space in real time, wherein the virtual shooting environment includes an immovable virtual background and the movable virtual shooting device.
[0262] Note 18. The display device according to Note 16 or 17, wherein:
[0263] At least a portion of the virtual background observed by the virtual shooting device is displayed on the display device as the virtual background picture, and
[0264] The virtual background picture is dynamically updated based on the position and / or posture of the virtual shooting device, thereby being dynamically updated based on the position and / or posture of the shooting device.
[0265] Note 19. The display device according to any one of Notes 13 to 18, wherein the position to which the first mobile device is to be moved is calculated based on the position and / or posture of the photographing device.
[0266] Note 20. A display device according to any one of Notes 13 to 19, wherein the display device is arranged on the first mobile device via a connecting component, and the weight of the display device is borne by the first mobile device and a spherical wheel located at the bottom of the connecting component.
[0267] Supplement 21. The display device according to Supplement 20, wherein the spherical wheel is a driven wheel that can move in various directions.
[0268] Note 22. The display device according to any one of Notes 13 to 21, wherein the first mobile device is a mobile robot.
[0269] Note 23. A shooting system comprising:
[0270] A display device that displays a virtual background image, and
[0271] A shooting device capable of moving around a shooting object to capture at least one image including the shooting object and the virtual background picture,
[0272] The photographed object is located between the photographing device and the display device.
[0273] The display device is arranged on the first mobile device and can be moved by the first mobile device so that the virtual background image has at least one of a predetermined angle, size and position in the image captured by the shooting device, and
[0274] The virtual background image is dynamically updated based on the position and / or posture of the camera.
[0275] Note 24. A shooting system according to Note 23, wherein the virtual background image has at least one of a predetermined angle, size and position in the image captured by the shooting device, including: the proportion of the virtual background image in the image captured by the shooting device is greater than a predetermined proportion threshold.
[0276] Supplement 25. The photographing system according to Supplement 24, wherein:
[0277] The proportion of the virtual background image in the image captured by the shooting device is greater than a predetermined proportion threshold, including:
[0278] The center line of the field of view of the camera device forms a predetermined angle with the display surface of the display device and passes through the longitudinal center line of the display surface, wherein the longitudinal center line divides the display surface into equal parts in the vertical direction, in particular, the predetermined angle is 90 degrees, and / or
[0279] The distance between the display device and the shooting device is less than a predetermined distance threshold, so that the degree of overlap between the left and right ends of the display surface and the left and right ends of the picture captured by the shooting device is greater than a predetermined overlap threshold.
[0280] Note 26. The photographing system according to any one of Notes 23 to 25, wherein:
[0281] The shooting device sends the obtained relative position information between the shooting device and the first mobile device to the processing device, so that the processing device can construct a virtual shooting environment in three-dimensional space in real time based on the relative position information, wherein the virtual shooting environment includes an immovable virtual background and a movable virtual shooting device corresponding to the shooting device.
[0282] Note 27. A display device according to any one of Notes 23 to 25, wherein relative position information between a virtual shooting device corresponding to the shooting device and the first mobile device is used by a processing device to construct a virtual shooting environment in a three-dimensional space in real time, wherein the virtual shooting environment includes an immovable virtual background and the movable virtual shooting device.
[0283] Supplement 28. The photographing system according to Supplement 26 or 27, wherein:
[0284] At least a portion of the virtual background observed by the virtual shooting device is displayed on the display device as the virtual background picture, and
[0285] The virtual background picture is dynamically updated based on the position and / or posture of the virtual shooting device, thereby being dynamically updated based on the position and / or posture of the shooting device.
[0286] Note 29. The photographing system according to any one of Notes 26 to 28, wherein:
[0287] The shooting device sends the obtained RGB information and / or depth information about the shooting object to the processing device, so that the processing device obtains a three-dimensional model reflecting the position and / or outline of the shooting object in the virtual shooting environment based on the RGB information and / or the depth information.
[0288] Supplement 30. The photographing system according to any one of Supplements 26 to 29, wherein:
[0289] The shooting device is held by a mechanical arm fixed at the bottom.
[0290] The photographing device is fixed to the end part of the robot arm, and
[0291] The fixing component between the photographing device and the end component of the robot arm makes the photographing center line of the photographing device parallel to the center line of the end component.
[0292] Supplement 31. The photographing system according to Supplement 30, wherein:
[0293] The bottom end of the mechanical arm is fixed to the second mobile device, and
[0294] The mechanical arm is coupled to the second moving device via a lifting mechanism for lifting the mechanical arm, and the mechanical arm is coupled to the lifting mechanism via a fixing component.
[0295] Note 32. The photographing system according to Note 30 or 31, wherein:
[0296] The movement of the robot arm and / or the second mobile device is calculated based on the pre-specified movement of the virtual camera, and the movement of the camera is determined based on the calculated movement of the robot arm and / or the second mobile device.
[0297] Note 33. The photographing system according to Note 30 or 31, wherein:
[0298] The photographing device sends the obtained image of the photographing object to the processing device, so that the processing device can identify the position and / or posture of the photographing object based on the image to determine the photographing position and / or moving route of the virtual photographing device, thereby determining the photographing position and / or moving route of the photographing device.
[0299] Note 34. The photographing system according to any one of Notes 23 to 33, wherein the position to which the first mobile device is to be moved is calculated based on the position and / or posture of the photographing device.
[0300] Note 35. A shooting system according to any one of Notes 23 to 34, wherein the display device is arranged on the first mobile device via a connecting component, and the weight of the display device is borne by the first mobile device and a spherical wheel located at the bottom of the connecting component.
[0301] Supplement 36. The photographing system according to Supplement 35, wherein the spherical wheel is a driven wheel that can move in various directions.
[0302] Note 37. The shooting system according to any one of Notes 31 to 33, wherein at least one of the first mobile device and the second mobile device is a mobile robot.
[0303] Note 38. The photographing system according to any one of Notes 23 to 37, wherein:
[0304] The shooting system also includes an anti-misoperation component, which prevents the first mobile device from moving quickly at a speed exceeding a predetermined speed threshold when it detects that the change in the position and / or posture of the shooting device exceeds a predetermined change threshold and / or the change rate of the position and / or posture of the shooting device exceeds a predetermined change rate threshold, and the center line of the field of view of the shooting device exceeds the range of the display surface of the display device.
[0305] Note 39. The photographing system according to any one of Notes 23 to 38, wherein:
[0306] The shooting system also includes a lighting component, which provides a fill light source for shooting by the shooting device, and sends its position information to the processing device for reproducing the fill light source in the virtual shooting environment of the three-dimensional space constructed by the processing device.
[0307] Supplement 40. The photographing system according to Supplement 39, wherein:
[0308] The photographing system further includes a shadow removing component for removing a shadow of the photographing object caused by the lighting component and appearing on the display surface of the display device.
[0309] Supplement 41. The photographing system according to Supplement 40, wherein:
[0310] The shadow elimination component estimates the corresponding area of the shadow of the object on the display surface based on the position information of the lighting component, the image information of the object obtained by the shooting device, and the relative position information between the shooting device and the display device, and performs shadow compensation processing on the corresponding area, thereby eliminating the shadow on the captured virtual background picture.
[0311] Supplement 42. The photographing system according to Supplement 40 or 41, wherein:
[0312] The first mobile device calculates a reflection area of the lighting component that reflects light on the shooting device due to specular reflection of the display surface during shooting based on the light source direction of the lighting component, the material of the display surface, and the ambient light; and
[0313] The photographing device moves away from the light reflecting area.
[0314] Note 43. The photographing system according to any one of Notes 26 to 42, wherein:
[0315] At least a part of the processing device is arranged in at least one of the photographing device, the display device, the first mobile device, and a server associated with the photographing system.
Claims
1. A photographing device, comprising: a processing circuit configured to control so that the shooting unit included in the shooting device moves around the shooting object to capture at least one image including the shooting object and a virtual background picture displayed on the display device, The photographed object is located between the photographing device and the display device. The display device is arranged on the first mobile device and can be moved by the first mobile device so that the virtual background image has at least one of a predetermined angle, size and position in the image captured by the shooting device, and The virtual background image is dynamically updated based on the position and / or posture of the camera.
2. The photographing device according to claim 1, wherein: The virtual background picture has at least one of a predetermined angle, size and position in the picture captured by the shooting device, including: the proportion of the virtual background picture in the picture captured by the shooting device is greater than a predetermined proportion threshold.
3. The photographing device according to claim 2, wherein: The proportion of the virtual background image in the image captured by the shooting device is greater than a predetermined proportion threshold, including: The center line of the field of view of the camera device forms a predetermined angle with the display surface of the display device and passes through the longitudinal center line of the display surface, wherein the longitudinal center line divides the display surface into equal parts in the vertical direction, in particular, the predetermined angle is 90 degrees, and / or The distance between the display device and the shooting device is less than a predetermined distance threshold, so that the degree of overlap between the left and right ends of the display surface and the left and right ends of the picture captured by the shooting device is greater than a predetermined overlap threshold.
4. The photographing device according to any one of claims 1 to 3, wherein: The processing circuit is configured to send the obtained relative position information between the shooting device and the first mobile device to the processing device, so that the processing device can construct a virtual shooting environment in a three-dimensional space in real time based on the relative position information, wherein the virtual shooting environment includes an immovable virtual background and a movable virtual shooting device corresponding to the shooting device.
5. The display device according to any one of claims 1 to 3, wherein: The relative position information between the virtual shooting device corresponding to the shooting device and the first mobile device is used by the processing device to construct a virtual shooting environment in three-dimensional space in real time, wherein the virtual shooting environment includes an immovable virtual background and the movable virtual shooting device.
6. The photographing device according to claim 4 or 5, wherein: At least a portion of the virtual background observed by the virtual shooting device is displayed on the display device as the virtual background picture, and The virtual background picture is dynamically updated based on the position and / or posture of the virtual shooting device, thereby being dynamically updated based on the position and / or posture of the shooting device.
7. The photographing device according to any one of claims 4 to 6, wherein: The processing circuit is configured to send the obtained RGB information and / or depth information about the photographed object to the processing device, so that the processing device obtains a three-dimensional model reflecting the position and / or outline of the photographed object in the virtual photographing environment based on the RGB information and / or the depth information.
8. The photographing device according to any one of claims 4 to 7, wherein: The shooting device is held by a mechanical arm fixed at the bottom. The photographing device is fixed to the end part of the robot arm, and The fixing component between the photographing device and the end component of the robot arm makes the photographing center line of the photographing device parallel to the center line of the end component.
9. A display device, comprising: a processing circuit configured to control so that a display unit included in the display device displays a virtual background picture, so that a shooting device capable of moving around a shooting object can capture at least one image including the shooting object and the virtual background picture, The photographed object is located between the photographing device and the display device. The display device is arranged on the first mobile device so as to be moved by the first mobile device, so that the virtual background image has at least one of a predetermined angle, size and position in the image captured by the shooting device, and The virtual background image is dynamically updated based on the position and / or posture of the camera.
10. A shooting system, comprising: A display device that displays a virtual background image, and A shooting device capable of moving around a shooting object to capture at least one image including the shooting object and the virtual background picture, The photographed object is located between the photographing device and the display device. The display device is arranged on the first mobile device and can be moved by the first mobile device so that the virtual background image has at least one of a predetermined angle, size and position in the image captured by the shooting device, and The virtual background image is dynamically updated based on the position and / or posture of the camera.
