Method for locking image capture frames and processing images

Abstract

A computer-implemented method for capturing photographic or video images comprises the steps of receiving a stream of data on a digital device and processing the data stream using machine learning methods, capturing an image in an automatic, semi-automatic or tracking mode, with the possibility of activating an image capture guide mode in which three-dimensional image capture points are generated in real time in an interactive map for each user individually. Each three-dimensional point contains interactive information for achieving a professionally composed frame. Said map is a two-dimensional map or an AR map and / or a mixed reality map. A mixed reality representation shows the three-dimensional image capture points on a connected mixed reality retransmission device and transmits guide signals for graphical output onto viewing screens of mixed reality devices, guiding the user to image capture points for favourable angles, and suggests how and in which position to hold the image capture device. The invention makes it possible to achieve professionally composed photographic or video frames without subsequent processing.

Description

[0001] METHOD OF CAPTURE OF SHOOTING FRAMEWORK AND PROCESSING IMAGES

[0002] AREA OF TECHNOLOGY

[0003] This technical solution pertains to the field of artificial intelligence (hereinafter referred to as AI), the task of which is the most accurate reproduction of the work of the human brain in the processes of decision-making, perception and definition of visual objects, images, compositions and other creative aspects that are intuitively understandable to humans. Such solutions function thanks to the principles and algorithms of machine learning, including deep machine learning.

[0004] In particular, the technical solution relates to the field of computer technology, in particular, to a method of preliminary, intuitive and automatic recording of frames (shooting) and image processing with the ability to target user traffic, using AI systems.

[0005] LEVEL OF TECHNOLOGY

[0006] The prior art discloses information source US 2019208117 A1, 04.07.2019, which discloses a method for providing recommendation information for creating photographs. The method includes detecting a subject's face on a preview screen viewed by a camera of the electronic device and displayed on the display of the electronic device by an electronic device, identifying information on the current composition of the preview screen based on the detected subject's face on the preview screen. A recommended photographic composition is determined based on the current information on the composition of the preview screen and information on the composition located in the center, and a visual composition guide is provided on the display of the electronic device, the movement of which is limited in one plane, based on a certain recommended photograph.Data processing occurs on the server, which means that to receive shooting recommendations, you need Internet access.

[0007] The declared technical solution differs from the known solution in that the interface does not contain elements that, when pressed, trigger shooting, i.e. the system itself determines the moment of pressing the "shutter button", and the interface can be tactile in a particular case to initiate the shooting process in automatic mode or in general - for service purposes. Panning and movement of the device in space is provided by the method in all available degrees of freedom and planes. Interaction with the results is recorded and processed by machine learning methods with reinforcement for subsequent improvement of the shooting processes and results. The declared technical solution uses the principle of decentralization of artificial intelligence calculations on devices without direct binding to servers (Al on the Edge).Thus, a significant part of the calculations and processes of processing incoming information is carried out by the system on the digital device itself, regardless of whether there is a network connection or not, and the obtained results of reinforcement learning, available for public exchange, can be synchronized directly with other devices via an open inter-network channel (for example, via a Wi-Fi channel without access to the Internet) or via the cloud when a connection is available.In addition, the claimed method contains a guided shooting mode, in which three-dimensional shooting points are generated in an interactive map in real time for each user individually, wherein each point contains interactive information to obtain a professionally composed frame, and the map is a two-dimensional map or an AR map and / or a mixed reality map, wherein the mixed reality representation displays three-dimensional shooting points on a connected mixed reality relay device and wirelessly transmits guiding signals of graphic output to the projection screens of mixed reality devices, directing the user to shooting points of good angles and prompting how and in what position to hold the shooting device while the algorithm shoots the frame.

[0008] ESSENCE OF THE INVENTION

[0009] The technical problem that the claimed technical solution is aimed at solving is the creation of a computer-implemented method of photographing or video shooting with a digital device that includes at least one optical device, based on the provision of recommendations for professional framing that does not require subsequent selection and post-processing.

[0010] The technical result consists in obtaining professionally composed photo or video frames, without subsequent post-processing. Further and everywhere, a frame means both a static photo frame and a video clip.

[0011] In a preferred embodiment, a computer-implemented method for photographing or video recording with a digital device, including at least one optical device, based on providing recommendations for professional framing, is claimed, comprising the steps of: receiving a data stream on the digital device through the optical device; processing the data stream using machine learning methods using models trained on a plurality of images, including at least composition information, in order to recommend the best frame; shooting is performed in automatic mode, semi-automatic mode, tracking shooting mode and in guided shooting mode; wherein, in automatic mode, when any area of ​​the screen is pressed, the shooting process is initiated; in semi-automatic or tracking shooting mode, shooting is performed while combining the frame capture zone and the three-dimensional frame of the frame;if the ability to create a professionally composed frame is not available in the automatic, semi-automatic and tracking modes, then the guide shooting mode is activated, in which three-dimensional shooting points are generated into an interactive map in real time for each user individually, with each point containing interactive information to obtain a professionally composed frame, and the map is a two-dimensional map or an AR map and / or a mixed reality map; the mixed reality representation displays three-dimensional shooting points on a connected mixed reality relay device and wirelessly transmits graphical output guide signals to the projection screens of mixed reality devices, directing the user to shooting points of good angles and prompting how and in what position to hold the shooting device while the algorithm shoots the frame;the frames obtained during shooting, in at least one of the modes, are sent to a buffer of pre-saved shooting results, where express post-processing is performed and the obtained shooting result is displayed on the screen; further interaction with the results is recorded and processed using machine learning methods with reinforcement for subsequent improvement of the shooting processes and results.

[0012] In a particular case, the optical device is a camera.

[0013] In another particular embodiment, the optical device includes at least one camera.

[0014] In another particular embodiment, the data stream contains at least metadata including EXIF.

[0015] In another particular embodiment, an individual portrait profile is created by scanning the face and at least one successful posing option is determined. In another particular embodiment, when a living object corresponding to the configured individual portrait profile is determined in the frame, the parameters of this profile will be taken into account during shooting.

[0016] In another particular variant, when more than one living object is detected in the frame, the living object with a saved portrait profile is selected first.

[0017] In another particular variant, in the automatic shooting mode, when pressing on any part of the screen, hold your finger and move the digital device in different planes and directions.

[0018] In another particular variant, the shooting result is displayed on the screen when the finger is released.

[0019] In another particular variant, in the semi-automatic shooting mode, the three-dimensional frame is a fixed reference point.

[0020] In another particular variant, in the semi-automatic shooting mode, they orient themselves relative to a three-dimensional frame that has been moved and fixed in space, and combine the frame capture zone and the three-dimensional frame of the frame.

[0021] In another particular variant, in the semi-automatic shooting mode and the tracking shooting mode, when a living object is determined in the focus coverage area, it is highlighted with a contour, then a three-dimensional mannequin is graphically drawn on top of the object and the mannequin animatedly changes its pose to a more successful one.

[0022] In another particular case, if more than one live object is defined, the user can select a priority object.

[0023] In another particular variant, in the tracking shooting mode, the three-dimensional frame is tracking.

[0024] In another particular variant, in the tracking shooting mode, the tracking frame capture zone and the tracking three-dimensional frame frame are stabilized relative to the central point and combined.

[0025] In another particular variant, a series of frames and / or a video sequence is taken a couple of seconds before and after stabilization and alignment.

[0026] In another particular variation, a 3D frame indicates the orientation of the camera to produce a professionally composed shot.

[0027] In another particular variant, the operator can select the guided shooting mode after starting the system.

[0028] In another particular variant, in the guided shooting mode, a map is launched, where the user selects the desired point, brings the device closer to this point and automatically switches the shooting mode to the one saved at this point. In another particular variant, a two-dimensional map with the display of three-dimensional shooting points can be displayed in full screen or as a thumbnail, the thumbnail can be moved to any place on the screen.

[0029] In another particular variant, the AR map indicates survey points taking into account the radius to them: within a radius of up to 50 meters, a three-dimensional object of the survey point is shown with detailed interactive information, such as: information about the rating of the point, information about the author of the point, information about the lifetime of the point; within a radius of 50 meters to 5 kilometers, three-dimensional objects of a cluster of survey points are shown with detailed information, such as: information about the number of points in the cluster, the distance to the cluster of points, and information about the lifetime of the cluster of points.

[0030] In another particular variant, the output of the shooting result occupies the central part of the screen, while the free areas of the screen remain active and when you press these active areas return to the mode during which the shooting was carried out.

[0031] In another particular variant, the obtained shooting result is put into a buffer until the next session of outputting the obtained shooting results, which are queued for sorting.

[0032] In another private option, to save the shooting result, drag the shooting result to the right.

[0033] In another particular variant, to delete the shooting result, drag the shooting result to the left.

[0034] In another particular case, when interacting with the results of the shooting, training can be carried out both on the device and on the server.

[0035] A machine-readable medium containing instructions executable by a processor, wherein the processor is configured to perform the steps of the above-described method.

[0036] In the context of this application, a system is understood to be a computing device that contains at least a processor and a memory, wherein the memory contains instructions that are executed by the processor. In general, a computing device contains components such as: one or more processors, at least one memory, a data storage means, input / output interfaces, an input / output means, and network interaction means.

[0037] The device processor performs the basic computing operations necessary for the functioning of the device or the functionality of one or more of its components. The processor executes the necessary machine-readable commands contained in the random access memory. The memory is usually implemented as RAM and contains the necessary software logic that provides the required functionality.

[0038] The data storage facility can be implemented in the form of HDD, SSD disks, RAID array, network storage, flash memory, optical storage devices (CD, DVD, MD, Blue-Ray disks), etc. The facility allows for long-term storage of various types of information, such as the above-mentioned files with user data sets, databases containing records of time intervals measured for each user, user identifiers, etc.

[0039] Interfaces are standard means for connecting and working with the server part, for example, USB, RS232, RJ45, LPT, COM, HDMI, PS / 2, Lightning, FireWire, etc.

[0040] The choice of interfaces depends on the specific design of the device, which can be a personal computer, mainframe, server cluster, thin client, smartphone, laptop, etc.

[0041] A keyboard must be used as a means of input / output of data in any embodiment of the system implementing the described method. The hardware implementation of the keyboard can be any known: it can be either a built-in keyboard used on a laptop or netbook, or a separate device connected to a desktop computer, server or other computer device. The connection in this case can be either wired, in which the keyboard connecting cable is connected to a PS / 2 or USB port located on the system unit of the desktop computer, or wireless, in which the keyboard exchanges data via a wireless communication channel, for example, a radio channel, with a base station, which, in turn, is directly connected to the system unit, for example, to one of the USB ports.In addition to the keyboard, the following may also be used as input / output means: joystick, display (touch display), projector, touchpad, mouse, trackball, light pen, speakers, microphone, etc.

[0042] Network interaction means are selected from a device that provides network reception and transmission of data, for example, an Ethernet card, WLAN / Wi-Fi module, Bluetooth module, BLE module, NFC module, IrDa, RFID module, GSM modem, etc. The means ensure the organization of data exchange via a wired or wireless data transmission channel, for example, WAN, PAN, LAN, Intranet, Internet, WLAN, WMAN or GSM.

[0043] The components of the device are connected via a common data transmission bus. The role of the device manipulator on which the method is implemented can be played by either a person - a user (operator) (hereinafter referred to as the operator), or a hardware and software complex for autonomous maneuvering and movement in space (hereinafter referred to as the HSC) of the device, made as an automated drone, UAV, robot, etc. (hereinafter referred to as the machine).

[0044] Application areas of the method: amateur photo and video shooting, professional camera shooting of photos and videos, computer vision, where a creative approach to image analysis is required, including medicine (ensuring the operation of promising eye implants), as well as such borderline applications as, for example, precise AR positioning or generation of photorealistic spaces for CP design, film and gaming industries, etc.

[0045] DESCRIPTION OF DRAWINGS

[0046] The implementation of the invention will be described further in accordance with the attached drawings, which are presented to explain the essence of the invention and in no way limit the scope of the invention. The following drawings are attached to the application:

[0047] FIG. 1 illustrates a storyboard of the process of interaction with the interface in the semi-automatic shooting mode.

[0048] FIG. 2 illustrates a storyboard of the process of interaction with the interface in the semi-automatic shooting mode, in the presence of a living object in the frame, for example, a person and / or an animal.

[0049] FIG. 3 illustrates the display in the guided shooting mode of an active map of shooting points (angles) in augmented reality (hereinafter referred to as the AR map).

[0050] DETAILED DESCRIPTION OF THE INVENTION

[0051] In the following detailed description of the embodiment of the invention, numerous implementation details are set forth in order to provide a clear understanding of the present invention. However, it will be apparent to one skilled in the art how the present invention may be used with or without these implementation details. In other instances, well-known methods, procedures, and components have not been described in detail in order not to unnecessarily obscure the features of the present invention.

[0052] In addition, it will be clear from the above description that the invention is not limited to the embodiment shown. Numerous possible modifications, changes, variations and substitutions, preserving the essence and form of the present invention, will be obvious to those skilled in the art.

[0053] In addition to the basic tasks of accurately determining objects, distance, depth, visual odometry and color correction, the claimed method solves the problem of correctly setting the frame, choosing the composition, exposure, point and angle of shooting (hereinafter - Perspective). Eliminates the need for sorting and post-processing images. And can also simulate the work of various optics.

[0054] For the operator, the method implemented by the system is a simple interaction functionality. When the operator points the camera of the digital device at the area of ​​interest for shooting, the system analyzes the available space around and begins to interact with the operator in four modes: automatic, semi-automatic, tracking and guided shooting. Each mode is described in detail below in the application materials.

[0055] The method contains stages in which the system automatically takes shots for the operator. The interface of the system that implements the method does not have the usual shutter release button.

[0056] Recognize objects in the frame, highlight the main one, determine the genre, successful composition, exposure, settings and color correction, point and angle. At the same time, track external atmospheric (weather, insolation, astronavigation) and time parameters. All this data is collected for processing and setting the frame. Synchronize all image streams from available cameras on a digital device. Determine a perspective frame for shooting, taking into account the movement of the device and the operator with the device in all planes with the maximum available angle.

[0057] During the process of camera aiming, shooting and processing at the software level, the operation of camera lenses with different focal lengths that are not available through hardware is simulated (computer optical correction).

[0058] The digital devices are designed to use the principle of edge computing AI (hereinafter referred to as edge AI), also known as "Edge A1" or "A1 on the Edge" (at the time of disclosure of information in the field of AI, there is no established term). When a significant part of the calculations and processes of processing incoming information occurs on the digital device itself. Edge AI does not require a network connection and / or an Internet connection, that is, it is possible to work without an Internet connection.

[0059] In this case, the exchange between digital devices, within the network, can be carried out by direct synchronization with other devices via open inter-network channels, such as Bluetooth, Wi-Fi, etc. without access to the Internet, or via the Internet directly or via the cloud (hereinafter referred to as the Server) the exchange method is determined automatically, making a choice in favor of maximum throughput. Digital devices exchange the obtained results, including the results of machine learning with reinforcement (necessary for continuous comprehensive improvement of the system), available for public exchange within the network.

[0060] To ensure the growth of the speed of calculations in real time, the possibility of interaction with other devices included in the system network is provided for in order to distribute computing power between them on demand (hereinafter referred to as fog computing).

[0061] The system, each time interacting with the operator, trains and adapts to the operator's preferences and tastes, becoming an individual tool and developing a unique shooting style. At the same time, the system also trains globally within the entire network to improve the algorithm's operation. And if the user limits the transfer of data on interaction with the system for training personalized improvements, then he will be able to use only available global improvements.

[0062] Any words in the text that are given in the singular may also be read, interpreted and construed as words with the same meaning in the plural, unless the context clearly indicates an unambiguous interpretation of the word in the singular.

[0063] The term operator refers not only to the user of the system and / or the owner of the device on which the system is installed, but also to any other person to whom the user and / or owner has entrusted the device to carry out filming.

[0064] To describe the claimed method, the most common at the time of disclosure of information variant of manipulation of the device is presented - "System and Operator". However, all the same described rules, algorithms and examples are applicable for the combination - "System and Machine".

[0065] The frame proportions directly depend on the physical proportions of the camera matrix. The standard value for most cameras is 4:3. The system at the software level, by cropping, can use, according to the user's preferences, any other frame proportions, for example, 3:2, 16:9, 1:1 or arbitrarily set values ​​in the photo settings.

[0066] The device is any digital device or machine equipped with cameras. The device must have a central processor, a graphics processor, RAM, special chips / modules for wireless remote and intranet data exchange. According to the operator's choice in the system settings, indicating whether he is right-handed or left-handed, all tactile interface elements that need to be pressed and / or held are "mirrored" from right to left or vice versa, providing a better user experience of interaction with the system.

[0067] The term labeled dataset is used to mean a set of frames in which all available objects are manually or programmatically marked graphically (with color and / or edging) and / or otherwise signed, as well as any other set of labeled data. Such a set is usually used in machine learning to train neural networks with the subsequent receipt of ready-made neural network models. Moreover, the success of machine learning directly depends on the volume of the initial information: the more of it, the better the AI ​​will develop.

[0068] To begin the description of the method of photographing or video shooting with a digital device that includes at least one optical device, based on the provision of recommendations for professional framing, it is worth considering the main basic optical capabilities of the devices using the system and explaining the possibilities of spatial interaction of the device system with the environment within the framework of the shooting process.

[0069] The number of cameras in a device is directly related to the capabilities of the form factor of the device itself, its compactness and the thickness of the camera lens mount. Lenses can have a fixed or variable focal length. Unlike fixed lenses, variable lenses have a complex design that requires telescopic extension of the lenses inside the lens to change the focal length. Compact devices are often equipped with one or a series of cameras with fixed lenses due to the overall capabilities of the devices. This includes cameras located on any side (front, back or side) of the device.

[0070] The single-camera option is usually, but not necessarily, a standard lens with a wide-angle equivalent focal length of about 26mm (angle of view of about 85°), closest to the focal length of the human eye.

[0071] The dual-camera version is usually, but not necessarily, equipped with a standard and a zoom lens, with a standard focal length equivalent of 52 mm (viewing angle of about 47 °), there are also combinations of two standard, a standard and an extended lens, with an ultra-wide focal length equivalent of 13 mm (viewing angle of about 120 °), etc.

[0072] A three-camera option, where all three types of lenses are used. A larger number of lenses, usually intermediate focal length options or promising options with wider coverage or longer focal length.

[0073] To analyze the surrounding situation and make decisions on composition and exposure, the method uses all the device's cameras with available frame coverage areas.

[0074] The coverage area (field of view) and the frame size that can be captured may vary depending on the selected focal length of the camera lens.

[0075] Thus, the field of view of a wide lens frame covers a larger shooting area, compared to the human gaze, and the effect of distance is obtained.

[0076] The field of view of a standard lens covers a smaller shooting area and, compared to the human gaze, the resulting image is very close in perception.

[0077] The field of view of a zoom or telephoto lens covers an even smaller shooting area compared to the human gaze, resulting in a zooming effect.

[0078] Alternative options are also possible, if there is hardware capability to accommodate a larger number of fixed lenses and / or use lenses with variable focal length, and it is also applicable to changing the focal length at the software level, simulating the operation of different lenses and their focal lengths.

[0079] A wide lens's coverage area is most often used in landscape photography to capture the scale of the scene being photographed.

[0080] Depending on the operator's settings, the interface may display an indicator of the field of view (zoom in / out) of the frame. By pressing the special zone once, the system will switch between the available lenses of the device. By pressing and holding the special zone, an animated circular scale will open. By moving this scale, you can manually select the required image magnification using the hardware capabilities of optical and digital zooming, including generative zooming.

[0081] The standard lens coverage area is used for multi-genre photography and is the most common.

[0082] A zoom or telephoto lens most often uses this coverage area in portrait photography or for shooting distant objects, including close-ups.

[0083] It is further worth noting that before and during the filming process, when the method is in operation, a variety of input data is analyzed. Such as images from cameras, geolocation data, data from a gyroscopic sensor, accelerometry data, visual odometry, and available terrain map relief. Thanks to the mentioned data, the process of moving the device and the operator in space is coordinated.

[0084] Thus, the system takes into account the dynamic panning technique "tracking". This technique is used to move the device in three planes taking into account the roll, yaw and pitch relative to the center of the device. That is, its movement and rotation along all three axes X, Y, Z.

[0085] The stationary panning technique of "looking around" the operator is also taken into account. This technique is used to freely move the device around the operator, with a limitation on collision with natural obstacles.

[0086] And also the dynamic panning technique "looking around" around the shooting object is taken into account. This technique is used for free movement of the operator together with the device around the shooting object, with a limitation on collision with natural obstacles.

[0087] That is, when making decisions about where and from what angle it is best to conduct perspective shooting, the system implementing the method is not limited only to the information entering the camera viewfinder, but thanks to spatial coordination can search for promising options for shooting that go beyond the boundaries of the coverage area and offer the operator to move to the desired shooting point.

[0088] Thus, the claimed method contains essential features that use visual information from all cameras with all available focal lengths simultaneously. At the same time, the system has no restrictions on the degrees of freedom for panning to create perspective shooting plans.

[0089] The system, as a comprehensive shooting solution, is mainly positioned as an individual solution that interacts with the device owner, learns and strives to meet the user's expectations regarding creative vision in shooting. However, it can also be used in multi-user solutions.

[0090] To improve the user experience of interaction with the system, the operator is offered to create an individual portrait profile at the moment of its first launch on the device. It is worth saying that if this step is skipped at the beginning, the user can return to it at any other time convenient for him.

[0091] Portrait profile is a combination of detailed scanning of the operator's face, selection of a favorable side and retouching settings that meet the user's expectations and are considered natural and the most favorable from the user's point of view for portrait shooting. The user is not limited to just one portrait profile; at his / her discretion, the required number of portrait profiles of other people can also be added to the system on the device, for example, close relatives, friends and any people in general. Scanning and fine-tuning of additional portrait profiles occurs exclusively with the consent of the operator.

[0092] The system is designed to provide maximum protection for data associated with portrait profiles, which are stored in encrypted form exclusively on the device itself, access to which is provided by a key-password or other user authentication systems available on the device, and are not synchronized either in the cloud or with other devices.

[0093] In case of complete loss of the device, individual portrait profiles will also be lost. Since this is the only type of data that cannot be synchronized when reinstalling the system on a new device. In this case, the user will be required to re-create the portrait profile on the new device.

[0094] However, the user can change this security setting in the settings and allow the system to sync portrait profiles to secure cloud storage.

[0095] To create a portrait profile, the operator films ("scans") his face using the front camera on the device he holds in his hands. Interface elements prompt the operator how to turn his head to perform an accurate scan of all the features of the head and facial features. The prompts are made in the form of interface elements, and sequentially indicate a full rotation of the head around, frontal and profile turns, lifting and tilting the chin, reproducing various emotions: sadness, neutral state, slight smile, confident smile, wide smile, laughter, closed eyes, usually open eyes, wide open eyes.

[0096] The AI ​​core of the system (hereinafter referred to as the Core) analyzes the received data and, by comparing it with professional portrait profiles that are part of the professional datasets of the system, determines the most successful posing option(s) for the user. The work of the core will be discussed further.

[0097] The user's task at the next stage is to select from the proposed options, in the "carousel" mode, the most accurate version of the "working" side of the portrait profile. Then adjust the parameters of the face and head. And save the result in the system. Now, during shooting, when they determine a living object in the frame that matches the configured portrait profile, the user's preferences for angle and retouching will be taken into account to form an ideal portrait frame.

[0098] This solves the problem when a user asks another operator to take a portrait photo and the result does not meet his expectations.

[0099] If the system detects more than one living object in the frame, with portrait profiles installed, the system will first select the portrait profile of the user / device owner as a priority, then the operator can independently select the main object and build the frame in accordance with its portrait profile, while trying as much as possible to match the other portrait profiles defined in the frame.

[0100] As part of the step-by-step algorithm for creating an individual portrait profile, the operator starts the portrait profile creation mode, then the first scanning pass is performed, points, features and characteristics are determined, the scanning is repeated, fixing and validating the data obtained for the first pass, and variants of the digital model of the operator's face / head are created for the subsequent selection of a suitable variant in the "carousel" and application of the anthropological features and retouching parameter settings. After that, the operator saves the resulting portrait profile and exits the mode.

[0101] The key and central element of the system is the AI ​​core of the system. The core's task is to process the incoming data flow in order to receive options for the composition and exposure of the frame at the output, figuratively speaking, to help the device see the world around it creatively, as a professional photographer and / or videographer would do.

[0102] So, initially, photo / video streams are collected from all available cameras of the device and, using an intermediate layer of pre-installed, trained convolutional neural network models (hereinafter referred to as CNNs), all objects are marked (hereinafter referred to as detected) in the frame streams, the frame depth and distance between objects are calculated, individual portrait profile(s) are detected (if available), the priority object in focus is determined, the illumination of the scene is assessed, and a single image matrix is ​​compiled from all cameras with markings by coverage zones.

[0103] Then the following data is collected from the device: from the gyroscope about the position in space, from the GPS / GLONASS sensors about geo-positioning, as well as accelerometry and visual odometry, if necessary, when the first two types of data are not enough, for example, when manipulating the Machine via the PAC. Next, the image metadata is unloaded from the streams, which include, but are not limited to, the following data describing the conditions and methods of obtaining, authorship, etc.: manufacturer of the digital device (camera), model of the digital device (camera), authorship, shutter speed, aperture, light sensitivity in ISO units, use of flash, frame resolution, focal length, matrix size, equivalent focal length, depth of field, date and time of shooting, camera orientation (vertical or horizontal), white balance type, exposure, histogram parameters, address of the shooting location, position in space, etc. (hereinafter referred to as EXIF ​​metadata).

[0104] If there is an Internet connection, data is loaded from open sources via API, including forecast data for the next 7 days in case of no Internet connection, on external atmospheric conditions (weather, insolation and astronavigation). Together, these data provide an accurate idea of ​​the scene illumination level and the duration of a certain state, and also help to accurately predict and select the correct composition and exposure, for example, for shooting during the "golden hour" (this is the first hour after sunrise and the last hour before sunset, although the exact duration varies depending on the time of year) or night slow-motion astrophotography of the Milky Way (the so-called "night timelapse"),

[0105] As a safety net, if no individual portrait profiles have been defined at this stage, all predefined individual portrait profiles are loaded into the overall process.

[0106] The core then supports the calculations by selecting an appropriate, trained neural network model from the database of trained neural models, a description of which will be presented in the analysis of the algorithm for the system's operation in automatic shooting mode.

[0107] During the continuous cycle of collecting the core data on the results of shooting for reinforcement learning, which will be described in detail within the description of the system's operation in terms of processing the result(s) of shooting, the core receives intermediate results of the neural models of machine learning with reinforcement based on the user's personalized data on the results and activity in social networks, the description of which will be presented in the operation of the system in the automatic shooting mode. Access to the mentioned data is provided by the user, at his own choice, and this choice can be changed at any time in the privacy settings of the system. As mentioned earlier, within the core's operation, at the final step of the deep machine learning algorithm, data on the shooting points on the AR map and their ranking are loaded to generate alternative options for composition and exposure.

[0108] Thus, during processing, the kernel can provide the system with three outputs:

[0109] - the ability to create a frame is present, composition and exposure options are provided;

[0110] - creation of a frame is likely, for which it is necessary to provide the system with more data (to expand the coverage area of ​​space);

[0111] - there is no possibility to create a high-quality, professionally staged shot, for example, when the shot is completely out of focus, in which case the system switches to the guided shooting mode, which will be described in detail below.

[0112] And in conclusion of the description of the AI ​​core of the system, it is worth pointing out that at the hardware level, to ensure the calculations of the edge AI, the system works together with the machine learning controller and automatically distributes tasks between the so-called “neural engine” system, the central and graphic processors.

[0113] At this point in the disclosure, all the basic principles and key elements of the system have been described and defined, allowing for a more complete understanding of the new method of photographing or video shooting with a digital device that includes at least one optical device, based on the provision of recommendations for professional framing.

[0114] After starting the system, the operator selects the shooting modes: automatic, semi-automatic, tracking and guided shooting mode. Below are examples of the above modes.

[0115] In automatic shooting mode, the operator only needs to point the camera at the desired object, press any place on the screen in the active zone and, holding it with the finger of the right or left hand (it doesn’t matter whether you are right-handed or left-handed), smoothly move the device in different planes and directions, according to the description of the panning methods above, at the user’s discretion.

[0116] At the moment of panning the frame, the incoming stream of images is actively captured together with metadata. At the same time, neural networks detect all objects in the frame, portrait profiles of users, if available, their priority, position and distance, frame depth, scene illumination, external atmospheric conditions (weather, insolation and astronavigation, etc.). After that, the AI ​​core of the system processes the input data and provides the system with options for the most successful combinations of composition and exposure. During automatic shooting, frames corresponding to successful combinations are sent to the buffer of pre-saved results, where express post-processing takes place.

[0117] And after the user releases his finger, the system displays the obtained result(s) on the interface screen for the user to judge.

[0118] The selection of results and a detailed description of the system's operating algorithm in terms of processing the survey result(s) will be discussed further in the text.

[0119] Going into the details of the method, it is worth analyzing the process of obtaining data on the corresponding trained neural network model.

[0120] The database (hereinafter referred to as the DB) of labeled datasets, from which the training process begins, is divided into three groups:

[0121] - genre datasets from selected professional shots published throughout the history of world photography;

[0122] - frames from social networks that pass the minimum validation threshold (depending on the genre, the threshold value of a certain validation parameter (likes, comments, reposts, clicks, transitions, etc.) may change (the process of determining the validation parameters is not disclosed for reasons of commercial confidentiality) are included in genre datasets with subsequent marking;

[0123] - genre datasets composed of frames of system users.

[0124] The database of labeled datasets is processed by machine learning methods designed to create trained models, including supervised learning methods, such as the previously mentioned CNNs, which are excellent at recognizing and defining images.

[0125] A model in machine learning (ML) is a set of artificial intelligence methods, the characteristic feature of which is not a direct solution to a problem, but learning in the process of applying solutions to many similar problems.

[0126] Trained to recognize images and their compositions depending on the shooting genres, neural models, as well as datasets, are distributed in a single database into three levels:

[0127] - models of professional composition (in the world of photo / video shooting there are over twenty well-known compositional rules and within the framework of disclosure of information about the declared method there is no need to talk about each one separately) and exposure of genre photography, which also include a group of professional portrait profiles, - models of popular composition and exposure of genre photography, trained on popular frames from social networks, models of user composition and exposure of genre photography, trained on continuously updated user frames.

[0128] Thus, the core is engaged in the selection of an appropriate, trained neural network model from the database of trained neural models, depending on the input data processed by the core.

[0129] At the same time, as mentioned earlier, so-called computer optical correction is used to simulate camera lenses that are not available on the device. For this purpose, trained models of generative adversarial networks (hereinafter referred to as GANs) are used, which generate frames that simulate the operation of lenses in their composition.

[0130] GAN consists of a Generator and a Discriminator neural network that iteratively compete with each other in the process of creating a realistic version of an image. The Generator seeks to create an imitation of distortion and approximation of objects that are distant from each other in the depth of the frame, corresponding to existing lenses, and the Discriminator, in turn, cuts off unrealistic versions that do not match the images produced by existing lenses.

[0131] GANs models are also accumulated in the core and are applied according to the scenario, at the stage of express post-processing, when the variant of the identified composition is intended for use outside the hardware focal length, and also when the operator wants to obtain a focal length different from the hardware one using software.

[0132] So, during the processing of the core, the system in automatic shooting mode can provide two outputs: the possibility of creating a frame is present, composition and exposure options are provided;

[0133] - there is no possibility to create a high-quality and professionally staged shot, for example, when the shot is completely out of focus. In this case, the system displays a warning on the screen that it is not possible to "catch" the shot under the previous conditions, and displays in the interface the options necessary for moving the device in space to capture more data (expand the coverage area). And if this does not help, then the system switches to the guided shooting mode, which will be described in detail below.

[0134] In the case of the first output, the core transfers to the system the generated variants of successful compositions and exposures, which are sent to the buffer of pre-saved results, where express post-processing takes place using genre settings (such as brightness, contrast, clarity, etc. - for all genres in the photo / video industry, industrial standards for the use of such parameters have already been developed, therefore, describing the declared method, within the framework of disclosure of information, we will not dwell on this) and, if necessary, using GANs (simulating lenses) and GANs (simulating artistic style).

[0135] The latest GANs have the same image generation engine as the lens-simulating GANs, only trained on the artistic styles / techniques of famous photographers and videographers, as well as popular photographers and video enthusiasts from social media.

[0136] After the operator releases his finger, the system displays the obtained result(s) on the interface screen for the user to judge.

[0137] The selection of results and detailed operation of the system, in terms of processing the shooting result(s), will be discussed and described further in the text.

[0138] The main thing that is worth noting during the detailed description of the shooting method is the logging (saving) of data on the saving and deletion of the shooting results by the user and data on the distribution of the result in social networks and the activity of interaction with it.

[0139] In this case, both types of data are logged with the consent of the user and at any time such consent can be revoked, and previously transmitted data, upon additional indication of this, can be deleted with the right to restore within a period not exceeding 30 days.

[0140] Both types of data are used to identify candidates for the status of "camera point", subsequent ranking and placement on the AR map, which will be discussed below, and for reinforcement learning globally within the network and the development of an individual approach to the user.

[0141] In this way, the system can adapt to the user's preferences and learn from its "mistakes" (when the shooting result is sent to the "Deleted" folder) in order to subsequently anticipate and guess the creative ideas of the user of the device on which the system is installed.

[0142] FIG. 1 illustrates the process of manipulating the device (100) in the semi-automatic shooting mode.

[0143] In order for the shooting to be performed in a semi-automatic mode, the operator must combine with each other (hereinafter referred to as “superimpose”), as successively demonstrated in frames 5 and 6 of FIGS. 1 and 2, the frame capture zone (1600) of the device (100) and the target three-dimensional frame of the frame (1601), which is freely displaced to the location of the target shooting point in the AR space, as successively demonstrated in frames 1, 2, 3, 4 of FIGS. 1 and 2.

[0144] At the moment of superposition, namely a couple of seconds before and a couple of seconds after, a series of frames and / or a video sequence is created, from which the result is selected.

[0145] FIG. 2, like FIG. 1, illustrates the process of manipulating the device (100) in the semi-automatic shooting mode, only in the indicated example a scene of shooting a living object (1700) in the frame, a person, is added. In this case, a living object (1700) is understood to mean not only a person, but also animals that can be recognized by CNNs.

[0146] When the system recognizes a living object (1700) in the coverage area in focus, it highlights it with a contour (1701), as successively demonstrated in frames 1, 2, 3 of FIG. 2. If more than one living object (1700) is recognized and highlighted with a contour (1701) in the coverage area, the user can select a priority object (1700) for shooting by pressing a finger on it on the screen of the device (100).

[0147] The system then graphically draws a three-dimensional mannequin (1702) over the object, as shown in frame 4 of FIG. 2.

[0148] And based on the recommendations received from the core, the mannequin (1702) animatedly changes its pose to a more successful one (1703), as consistently demonstrated in frames 4 and 5 of FIG. 2.

[0149] In addition to the above actions, to perform portrait photography in semi-automatic mode, the operator must advise the person - the subject of the shooting (1700) to take the proposed successful pose (1703) and / or try to catch the subject of the shooting (1700) when he / she takes a position corresponding to the proposed pose (1703).

[0150] It is worth noting that in the semi-automatic shooting mode, the priority is the overlap of the coverage area (1600) and the target 3D frame (1601), and only secondarily - the coincidence of the shooting object (1700) with a successful pose (1703). At the same time, the system will try to bring the shooting moment as close as possible when both tasks are completed.

[0151] At the moment of panning the frame and pointing the cameras at the desired shooting object, the incoming image stream is captured along with metadata. At the same time, CNNs detect all objects in the frame, portrait profiles of users, if available, their priority, position and distance, frame depth, scene illumination, external atmospheric conditions (weather, insolation and astronavigation, etc.). After that, the AI ​​core of the system processes the input data and provides the system with options for the most successful combinations of composition and exposure. The best option in the form of a three-dimensional object of the target frame (1601) is shifted to the location of the target shooting point in AR space, as sequentially demonstrated in frames 1, 2, 3, 4 of FIG. 1 and 12.

[0152] Going into the details of the method, it is worth noting that the processes of obtaining data on the corresponding trained neural network model coincide with the processes mentioned earlier. To simulate camera lenses that are not available on the device, so-called computer optical correction is used, which uses GAN models.

[0153] So, during the processing of the core, the system in semi-automatic shooting mode can provide three outputs:

[0154] - the ability to create a frame is present, composition and exposure options are provided;

[0155] - creation of a frame is likely, for which it is necessary to provide the system with more data (to expand the coverage area);

[0156] - there is no possibility to create a high-quality, professionally staged shot, for example, when the shot is completely out of focus, in which case the system switches to the guided shooting mode, which will be described in detail below.

[0157] In the case of the first output, the core transmits to the system the generated variants of successful compositions and exposures, the best of which, according to the ranking result, in the form of a three-dimensional object of the target frame (1601) is shifted to the location of the target shooting point in AR space, as sequentially demonstrated in frames 1, 2, 3, 4 of FIG. 1 and 2.

[0158] After the semi-automatic shooting process itself, described above, the obtained result is sent to the pre-saved results buffer, where express post-processing takes place.

[0159] The system then displays the obtained result(s) on the interface screen for the user to judge.

[0160] The selection of results and detailed operation of the system, in terms of processing the shooting result(s), will be discussed and described further in the text.

[0161] The main thing to pay attention to is data logging (saving), which was also described in detail above.

[0162] Next, let's look at another tracking mode.

[0163] In order for the shooting to be performed in the tracking mode, the operator must stabilize, relative to the central point of each other and "superimpose" one on the other the frame capture zone (1600) of the device and the tracking ("swinging") three-dimensional frame frame (1601), which is obvious to the operator, but not significantly from the point of view of the device interface, shifts relative to the frame center, in accordance with the principles of panning, in the direction where the final frame should be captured, corresponding to the most successful composition and exposure. That is, the "tracking" process is carried out to the virtual location of the target shooting point in the AR space, where the location of the target point is made without graphic representations on the screen, by minor shifts of the frame (1601).

[0164] At the moment of stabilization / centering and superposition, namely a couple of seconds before and a couple of seconds after, the system takes a series of frames and / or a video sequence, from which the system selects the result.

[0165] The principle of creating portrait shots and / or other genre photography of living objects (1700) within the tracking mode is fully consistent with the principle described above, applicable to semi-automatic shooting.

[0166] Thus, at the moment of panning the frame and pointing the cameras at the desired shooting object, the system actively captures the incoming image stream together with metadata. At the same time, CNNs detect all objects in the frame, portrait profiles of users, if available, their priority, position and distance, frame depth, scene illumination, external atmospheric conditions (weather, insolation and astronavigation, etc.). After that, the AI ​​core of the system processes the input data and provides the system with options for the most successful combinations of composition and exposure. The best option is loaded in the background to the location of the target shooting point in the AR space, and is an invisible reference point for shifting the tracking (“swinging”) three-dimensional frame frame (1601), as consistently demonstrated in frames 1, 2, 3, 4, 5, 6 of FIG. 1.

[0167] After the tracking shooting process itself, the obtained result(s) are sent to the pre-saved results buffer, where express post-processing takes place.

[0168] The system then displays the obtained shooting result(s) on the interface screen for the user to judge.

[0169] The selection of results and a detailed algorithm for the system's operation in terms of processing the survey result(s) will be discussed further in the text.

[0170] Going into the details of the method, it is worth noting that the processes of obtaining data on the corresponding trained neural network model coincide with the processes mentioned earlier. To simulate camera lenses that are not available on the device, so-called computer optical correction is used, which uses GAN models.

[0171] So, during the processing of the core, the system in the tracking shooting mode can provide three outputs: - the possibility of creating a frame is present, composition and exposure options are provided;

[0172] - the creation of a frame is likely, for which it is necessary to provide the system with more data (to expand the coverage area), and visually the frame is tilted and / or shifted relative to the center of the frame, indicating the direction for expanding the field of view of the cameras;

[0173] - there is no possibility to create a high-quality, professionally staged shot, for example, when the shot is completely out of focus, in which case the system switches to the guided shooting mode, which will be described in detail below.

[0174] In the case of the first output, the core transmits to the system the generated variants of successful compositions and exposures, the best of which, based on the ranking result, is moved in the background to the location of the target shooting point in the AR space, and serves as an invisible reference point for shifting the tracking (“swinging”) three-dimensional frame of the frame.

[0175] After the tracking shooting process itself, the obtained result(s) are sent to the pre-saved results buffer, where express post-processing takes place, the process of which is described in detail above.

[0176] The system then displays the obtained result(s) on the interface screen for the user to judge.

[0177] The selection of results and a detailed block diagram of the system's operation algorithm in terms of processing the survey result(s) will be discussed and described further in the text.

[0178] The main thing to pay attention to is data logging (saving), which was also described in detail above.

[0179] Thus, all the above-described shooting modes of the disclosed method go through the same cycle of the shooting process.

[0180] Next, let's look at the fourth shooting mode of the declared method - the guided shooting mode.

[0181] FIG. 3 illustrates the system interface in the guided shooting mode with an active map of three-dimensional shooting points (angles) (2400) in augmented reality (hereinafter referred to as the AR map).

[0182] The user has access to three types of display of survey points (2400) in the interface:

[0183] The first type is a two-dimensional map (2410), which can be displayed in full screen or as a thumbnail (2407), which is located at the bottom of the screen by default, but the user can drag it to any convenient place to provide better navigation by shooting points (2400). The second type is an AR map projected onto the device screen (100) in the form of three-dimensional graphic and text elements (2400, 2401, 2402, 2403, 2404, 2405, 2406, 2407, 2408, 2409, 2410), complementing the real space that falls into the camera lenses.

[0184] The third type is a mixed reality map, the mixed reality representation displays three-dimensional shooting points on the connected mixed reality relay device (105) and wirelessly transmits graphic output guidance signals (2411) to the projection screens of the mixed reality devices (105), directing the user to shooting points of good angles and prompting how and in what position to hold the shooting device while the algorithm shoots the frame.

[0185] Thus, in FIG. 3 the following important elements are presented in the form of three-dimensional objects placed in augmented reality.

[0186] The shooting point (angle) (2400) is a three-dimensional object that can be made in the form of a three-dimensional frame, a digital device with a screen, a photo card, a baguette frame, etc. The user can select display types in the system settings, and a simple digital frame is used by default. The shooting point (2400) exactly corresponds to the coordinates and position of the frame in space made by the user (hereinafter referred to as the author), not always by the user / operator of the device (100).

[0187] The points (2400) in their structure consist of metadata about the angle and position of the device (100) in space, the camera settings at the time of shooting, as well as the level of social recognition of this point (the number of “repeat” shots taken at this point), which allows for the graphic recreation of three-dimensional objects.

[0188] Points (2400) are displayed in augmented reality if they are within a radius of 50 meters from the operator. This parameter is set by default and can be changed by the user at any time up to 1000 meters in the settings.

[0189] The following data is displayed inside the point (2400): the public name of the point's author (2406), the organic rating of the point (2405) - this is the number of photos that were taken (saved in the device's memory (100) and / or posted on the Internet) by other users of the system, when interacting with the selected point, and the lifetime of the point (2403) - this is the remaining time of the point's operation, taking into account the current data measured by the device (100), as well as the downloaded forecast data.

[0190] By clicking on the dot (2400), you can view the original frame that was taken by the author (2406), posted on social networks, view a brief overview of the author's account (2406), his rating in the system, other frames, subscribe (follow) or send a request to the chat for correspondence.

[0191] A group (cluster) of points (2401) is a three-dimensional object that can be made in the form of a three-dimensional ball, a plate, a cube, and other figures with a pointer perpendicular to the earth's surface. The user can select display types in the system settings, and a simple ball shape is used by default. Groups of shooting points (2401) are clusters of points (2400) in one place, which are displayed in augmented reality if they are within a radius of 50 meters to 10 kilometers from the operator. The first parameter of the near boundary of the radius is set by default, and can be changed by the user at any time starting from 1000 meters in the system settings.

[0192] Inside the point group object (2401), the number of points (2404) in the group is displayed as a number, and the distance from the operator to the point group (2402) is indicated next to it. Depending on the system of measures and weights selected in the settings, metric or imperial, the distance will be displayed in kilometers or miles. The lifetime of the point group (2403) is also displayed next to it - this is the remaining operating time of the fastest point (2400) inside the group (2401), taking into account the current data measured by the device (100), as well as the downloaded forecast data.

[0193] After the lifetime (2403) of the earliest point has expired, the displayed number of points (2404) will decrease by one, and the lifetime (2403) of the point group (2401) will be updated to the nearest lifetime of the next point (2400) in the group (2401).

[0194] Regardless of the radius of the position, all points can be found on a two-dimensional map (2410). There you can also see a point in the center (2409), displaying the exact location of the device (100) and the coverage area and direction (2408) where it (100) is "looking".

[0195] In popular places, where there are potentially more than 10 points (2400) within a radius of 50 meters, and there is more than one active user of the system in the zone of intersected kilometer radii, the system will show each user no more than ten points (2400) at a time, and these points (2400) will differ from each other and from the points (2400) that other users of the system see on the AR map.

[0196] The guiding signals of the graphic output (2411) on the projection screens of the mixed reality devices (105) are located in two optical perception zones: foveal and peripheral. The foveal guides prompt how and in what position to hold the shooting device while the algorithm shoots a frame. The peripheral guides, in turn, direct the user to the shooting points of the selected successful angles. After receiving confirmation from the AI ​​core about the impossibility of shooting in automatic, semi-automatic or tracking mode, the system switches the operator to the guided shooting mode and launches the AR map (FIG. 3). The operator searches for a suitable point (2400) for shooting, can additionally receive information about the point (2400) by clicking on it. After selecting the desired point (2400), the operator needs to bring the device (100) closer to the point (2400) at a distance of thirty centimeters, and then the system will switch to one of the shooting modes described earlier.Automatic, semi-automatic or tracking mode - depends on the point (2400) and / or the selected preferences in the user settings. By default, the tracking mode is selected for the transition. The shooting processes in these modes were discussed in detail earlier. And after the shooting, the results are saved / deleted, and the metadata about the sorting are logged and used for machine learning of the core with reinforcement. The guided shooting mode can also be used individually, the operator, after starting the system, can immediately select this mode.

[0197] The operation of the AR map (FIG. 3) of the shooting points (2400) with interactive interaction and elements of social activity and networking is supported by the central processor of the device (100), the server and a specialized software module for controlling the display of elements in augmented reality.

[0198] The AR map (FIG. 3) constantly monitors incoming data for the current display of points (2400) in augmented reality. Thus, EXIF, GPS / GLONASS, visual odometry and exposure metering data coming from the database of marked datasets provide an idea of ​​the position and initial illumination of a point in space. Data on the activity of social network users in interaction with the results of shooting published in social networks help to correctly rank points (2400) so that a promising point (2400) overcomes threshold values ​​​​in order to be able to be placed on the AR map (FIG. 3). Data on external atmospheric conditions (weather, illumination, astronavigation, etc.) are loaded from open sources of meteorological conditions, insolation, astronavigation via API, together with forecast values ​​​​for seven days in advance, so as not to depend on the Internet connection, to clarify the update of the lifetime of points (2403).

[0199] As described above, the points (2400), in the guided shooting mode, can be displayed both on a two-dimensional map (2410), where you can see all available points (2400) on the map, and in the augmented and mixed reality mode, but taking into account the radius from the operator: in a radius of up to 50 meters, it shows three-dimensional objects of shooting points (2400), with detailed interactive information about the point rating (2405), author (2406) and the point lifetime (2403); in a radius of 50 meters to 10 kilometers from the operator, it shows three-dimensional objects of point clusters (2401), with detailed information about the number of points (2404), the distance to them (2402) and the group lifetime (2403).

[0200] The following describes the process of user interaction with the interface during the selection of the obtained results.

[0201] The output of the shooting result in the system interface for saving or deleting the obtained shooting result occurs immediately after the system has completed the shooting and express post-processing, and can be displayed in the interface both in horizontal and vertical screen layout.

[0202] The result occupies the central area of ​​the screen. The remaining areas, at the edges of the screen, are active (clickable) and, when pressed once, instantly return the user to the current shooting mode, which continues to work in the background for another 10 seconds. At the same time, the previously obtained result is postponed in a special buffer folder inside the application until the next session of outputting the obtained results and is queued. The postponed results are not saved in the results folder on the device until the sorting procedure is completed, but are in a temporary buffer, so the photo feed is not clogged with unnecessary frames.

[0203] When saving the shooting result(s) to the photo feed, the user drags (swipes) the result to the right, thus giving the command to the system to save the provided result.

[0204] If there were several shooting results and / or the buffer folder has accumulated results that require sorting, then these results will appear one after another following the saved result.

[0205] When deleting a shooting result(s), the user drags (swipes) the result to the left, thus commanding the system to delete the provided result.

[0206] If there were several shooting results and / or the buffer folder has accumulated results that require sorting, then these results will appear one after another following the deleted result.

[0207] The deleted result is stored in the internal storage of the system inside the device and can be restored by the user within 30 days if it was deleted by mistake.

[0208] All metadata about the sorting of results is logged.

[0209] The system operation in terms of processing the shooting result(s) is discussed below. After saving / deleting the results, the data on saving and deleting the shooting results by the user and the data on dissemination of the result in social networks and the activity of interaction with it are logged (accounted for).

[0210] Both types of data are logged with the consent of the user and such consent can be revoked at any time, and previously transmitted data, upon additional indication of this, can be deleted with the right to restore within a period not exceeding 30 days.

[0211] Both types of data are used by the system: to identify candidates for the status of "shooting point", subsequent ranking and placement on the AR map (FIG. 3) and for reinforcement learning globally within the network and the development of an individual approach to the user. In this way, the system can adapt to the user's preferences and learn from its "mistakes" (when the shooting result is sent to the "Deleted" folder), and metadata about successful shots, as well as data on successful imitation of lenses, are added to user genre datasets in order to subsequently anticipate and guess the creative ideas of the user of the device on which the system is installed.

[0212] In this way, trained neural models and GANs are continuously updated, thereby improving the deep machine learning core, the shooting result and the user experience of interaction with the system, both globally within the network and personalized.

[0213] Next, we will consider the processes and tasks that are processed on the server. The databases of marked datasets are constantly synchronized with devices, updated and replenished. At the same time, to simulate camera lenses that are not available on the device, so-called computer optical correction is used, which uses GANs models.

[0214] Data on their use in creating results is logged and synchronized with the labeled datasets.

[0215] Also, on the server, in addition to the constantly training AI Core, data on interaction with the results of shooting in social networks is collected and logged, and based on this data, the system is trained, which increases the user experience of interaction with shooting modes, including guided shooting in augmented reality.

[0216] Cloud computing is necessary for fundamental calculations and global reinforcement learning on data collected from all devices and other sources during the day. The server part constantly updates the core and this is the only way for those users who, guided by personal considerations of security and privacy, have prohibited data exchange with the system cloud, while they consciously refuse to personalize the system to their preferences.

[0217] In conclusion of the disclosure of the application, it is necessary to consider the key feature of the system, which is described within the general block diagram of the interaction of the AI ​​cores of the system on devices using edge AI technologies, their process of data exchange in the system network and the Internet and channels for distributing computing power between devices, using fog computing. Using edge AI technologies, all calculations within the internal cores are performed on the devices themselves (100), which in turn are able to directly establish data exchange channels, which ensures not only the speed of exchange of marked and processed data, but also allows using the possibilities of distributing computing power between devices, significantly accelerating the data processing process. At the same time, the structure also invariably includes a cloud core located on the system servers, access to which and exchange occurs via Internet channels.

[0218] Thanks to such a structure, as the network grows, users will be able to carry out filming of any complexity as autonomously and quickly as possible without delays.

[0219] The present application materials present a preferred disclosure of the implementation of the claimed technical solution, which should not be used as limiting other, particular embodiments of its implementation that do not go beyond the scope of the requested scope of legal protection and are obvious to specialists in the relevant field of technology.

Claims

FORMULA 1. A computer-implemented method for photographing or video recording with a digital device that includes at least one optical device, based on providing recommendations for professional framing, comprising the following stages: receiving a data stream on the digital device through the optical device; processing the data stream using machine learning methods using models trained on a plurality of images that include at least composition information in order to recommend the best frame; shooting is performed in automatic, semi-automatic, tracking, and guided shooting modes; wherein, in automatic mode, pressing any area of ​​the screen initiates the shooting process; in semi-automatic or tracking shooting mode, shooting is performed while aligning the frame capture area and the three-dimensional frame of the frame;if the ability to create a professionally composed frame is not available in the automatic, semi-automatic and tracking modes, then the guide shooting mode is activated, in which three-dimensional shooting points are generated into an interactive map in real time for each user individually, with each point containing interactive information to obtain a professionally composed frame, and the map is a two-dimensional map or an AR map and / or a mixed reality map; the mixed reality representation displays three-dimensional shooting points on a connected mixed reality relay device and wirelessly transmits graphical output guide signals to the projection screens of mixed reality devices, directing the user to shooting points of good angles and prompting how and in what position to hold the shooting device while the algorithm shoots the frame;the frames obtained during shooting, in at least one of the modes, are sent to a buffer of pre-saved shooting results, where express post-processing is performed and the obtained shooting result is displayed on the screen; further interaction with the results is recorded and processed using machine learning methods with reinforcement for subsequent improvement of the shooting processes and results.

2. The method according to claim 1, characterized in that the optical device is a camera.

3. The method according to claim 2, characterized in that it includes at least one chamber.

4. The method according to claim 1, characterized in that the data stream contains at least metadata, including EXIF.

5. The method according to item 1, characterized in that an individual portrait profile is created by scanning the face and at least one successful posing option is determined.

6. The method according to paragraph 5, characterized in that when determining a living object in the frame that corresponds to the configured individual portrait profile, the parameters of this profile will be taken into account during shooting.

7. The method according to item 6, characterized in that when more than one living object is determined in the frame, the living object with a saved portrait profile is selected first.

8. The method according to item 1, characterized in that in the automatic shooting mode, when pressing on any part of the screen, the finger is held and the digital device is moved in different planes and directions.

9. The method according to item 8, characterized in that the shooting result is displayed on the screen when the finger is released.

10. The method according to claim 1, characterized in that in the semi-automatic shooting mode the three-dimensional frame is a fixed reference point.

11. The method according to item 1, characterized in that in the semi-automatic shooting mode, they are oriented relative to the three-dimensional frame moved and fixed in space, and the frame capture zone and the three-dimensional frame of the frame are combined.

12. The method according to item 1, characterized in that in the semi-automatic shooting mode and the tracking shooting mode, when a living object is determined in the focus coverage area, it is highlighted with a contour, then a three-dimensional mannequin is graphically drawn on top of the object and the mannequin animatedly changes its pose to a more successful one.

13. The method according to claim 12, characterized in that if more than one living object is determined, the user can select a priority object.

14. The method according to claim 1, characterized in that in the tracking shooting mode the three-dimensional frame is tracking.

15. The method according to item 1, characterized in that in the tracking shooting mode, the tracking frame capture zone and the tracking three-dimensional frame frame are stabilized relative to the central point and combined.

16. The method according to paragraph 11 and paragraph 15, characterized in that a couple of seconds before and after stabilization and alignment, frames and / or a video series are shot.

17. The method according to paragraph 11 and paragraph 15, characterized in that the three-dimensional frame indicates the orientation of the camera, in order to obtain a professionally composed frame.

18. The method according to item 1, characterized in that the operator can select the guided shooting mode after starting the system.

19. The method according to item 1, characterized in that in the guided shooting mode, a map is launched, where the user selects the desired point, brings the device closer to this point and automatically switches the shooting mode to the one saved at this point.

20. The method according to item 1, characterized in that the two-dimensional map with the display of three-dimensional survey points can be displayed on the entire screen or as a thumbnail, the thumbnail can be moved to any place on the screen.

21. The method according to claim 1, characterized in that the AR map indicates survey points taking into account the radius to them: within a radius of up to 50 meters, a three-dimensional object of a survey point is shown with detailed interactive information, such as: information about the rating of the point, information about the author of the point, information about the lifetime of the point; within a radius of 50 meters to 5 kilometers, three-dimensional objects of a cluster of survey points are shown with detailed information, such as: information about the number of points in the cluster, the distance to the cluster of points, and information about the lifetime of the cluster of points.

22. The method according to item 1, characterized in that the output of the shooting result occupies the central part of the screen, while the free areas of the screen remain active and when these active areas are pressed, they return to the mode during which the shooting was carried out.

23. The method according to item 22, characterized in that the obtained shooting result is placed in a buffer until the next session of outputting the obtained shooting results, which are placed in a queue for sorting.

24. The method according to paragraph 23, characterized in that in order to save the shooting result, the shooting result is dragged to the right.

25. The method according to item 23, characterized in that in order to delete a shooting result, the shooting result is dragged to the left.

26. The method according to paragraph 1, characterized in that when interacting with the results of the shooting, training can be carried out both on the device and on the server.

27. A machine-readable medium containing instructions executable by a processor, wherein the processor is configured to perform the steps of the method according to paragraphs 1-26.

Images