Image processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an image processing method and device, electronic equipment and a storage medium, and is at least applied to the naked eye 3D field and the artificial intelligence field, and the method comprises the steps: collecting an environment video in a preset range of display equipment through an image collection device of the display equipment in a process that the display equipment displays a prefabricated video; in response to determining that a target object exists in the preset range based on the environment video and the current state of the target object meets a preset state index, acquiring a current playing picture of the prefabricated video; and based on the current playing picture, determining an interactive video matched with the current state of the target object, and displaying the interactive video on the display device. Through application of the method and the device, a deep and efficient interaction mode can be expanded in a process of playing the prefabricated video by the display equipment in a resource intensive mode, and the playing quality of the prefabricated video is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of naked-eye 3D technology, and relate to but are not limited to an image processing method, device, electronic device and storage medium. Background Art

[0002] A naked-eye 3D screen refers to a display system that uses the parallax characteristics of the human eyes to obtain a realistic three-dimensional image with space and depth without the need for any auxiliary equipment (such as 3D glasses, helmets, etc.).

[0003] However, the applicant discovered in the process of implementing this application that the naked-eye 3D screen in the related art is only used as a pure content material display without interactive experience. This traditional way of displaying pure content materials limits the depth of information interaction of naked-eye 3D screen viewers. Therefore, when viewers watch pre-made videos of pure content, if there is a need for further interaction with the content in the pre-made video, they have to use other means to further interact with the pre-made video played in the naked-eye 3D screen, or they need to play the pre-made video again through other devices for further interaction. Obviously, this is a huge waste of the resources of the naked-eye 3D screen (including the computing resources and communication resources of the background server).

[0004] In summary, there is no effective solution in the relevant technology on how to expand in-depth and efficient interactive methods in the process of playing pre-produced videos on naked-eye 3D screens in a resource-intensive manner and improve the playback quality of pre-produced videos. Summary of the invention

[0005] The embodiments of the present application provide an image processing method, device, electronic device and storage medium, which can at least be applied to the naked-eye 3D field and the artificial intelligence field, and can expand in-depth and efficient interactive methods in the process of playing pre-produced videos on a display device in a resource-intensive manner, thereby promoting the playback quality of pre-produced videos.

[0006] The technical solution of the embodiment of the present application is implemented as follows:

[0007] An embodiment of the present application provides an image processing method, including: in the process of displaying a pre-produced video on a display device, collecting an environmental video within a preset range of the display device through an image acquisition device of the display device; in response to determining that a target object exists within the preset range based on the environmental video and that the current state of the target object meets a preset state indicator, obtaining a current playback screen of the pre-produced video; based on the current playback screen, determining an interactive video that matches the current state of the target object, and displaying the interactive video on the display device.

[0008] An embodiment of the present application provides an image processing device, which includes: an acquisition model, which is used to acquire an environmental video located within a preset range of the display device through an image acquisition device of the display device during the process of displaying a pre-produced video; an acquisition module, which is used to acquire a current playback screen of the pre-produced video in response to determining that a target object exists within the preset range based on the environmental video and that the current state of the target object meets a preset state indicator; and a determination module, which is used to determine an interactive video that matches the current state of the target object based on the current playback screen, and display the interactive video on the display device.

[0009] In some embodiments, the acquisition module is also used to: perform video recognition on the environmental video to obtain a video recognition result for the preset range; if the video recognition result indicates that the target object is within the preset range, determine the number of the target objects; the number of target objects constitutes the current state of the target object; if the number of target objects is greater than a quantity threshold, determine that the current state of the target object meets the preset state indicator, and obtain the current playback screen of the pre-produced video.

[0010] In some embodiments, the acquisition module is also used to: perform video recognition on the environmental video to obtain a video recognition result for the preset range; if the video recognition result indicates that the target object is within the preset range, determine the action type of the target object; the action type of the target object constitutes the current state of the target object; if the action type of the target object is a preset type, determine that the current state of the target object meets the preset state indicator, and obtain the current playback screen of the pre-produced video.

[0011] In some embodiments, the acquisition module is also used to: perform video frame capture on the environmental video to obtain multiple video frames; perform content consistency analysis on each video frame with images in a preset image library to obtain a consistency analysis result; if the consistency analysis result between the video frame and any image in the preset image library is consistent, obtain attribute information corresponding to the image from the preset image library; determine a video frame recognition result corresponding to the video frame based on the attribute information; and summarize all video frame recognition results in the environmental video to obtain a video recognition result of the environmental video.

[0012] In some embodiments, the acquisition module is also used to: input the environmental video into a pre-trained video object quantity recognition model, identify the number of target objects in the environmental video through the video object quantity recognition model, and obtain the number of target objects for the preset range; input the environmental video into a pre-trained video action recognition model, identify the action of the target objects in the environmental video through the video action recognition model, and obtain the action type of the target objects for the preset range.

[0013] In some embodiments, the determination module is also used to: perform virtual object identification on the current playback screen to obtain virtual objects in the current playback screen; if the number of target objects is greater than a quantity threshold, determine an interactive action that matches the number of target objects for the virtual objects; obtain first rendering data for the virtual object to perform the interactive action; and use the first rendering data to perform image rendering to obtain the interactive video.

[0014] In some embodiments, the environmental video has a preset duration, and a segment of the environmental video is captured every preset duration; the device also includes: a video acquisition module, used to acquire the current environmental video within a current preset duration and a historical environmental video within a historical preset duration connected to the current environmental video and located before the current preset duration; a video recognition module, used to perform video recognition on the current environmental video and the historical environmental video, respectively, and correspondingly obtain the current video recognition result and the historical video recognition result; a quantity extraction module, used to extract the number of target objects from the current video recognition result and the historical video recognition result, respectively, and correspondingly obtain the current number and historical number of target objects; an expression change data determination module, used to determine the expression change data of the virtual object based on the current number and the historical number; a data adding module, used to add the expression change data to the first rendering data.

[0015] In some embodiments, the expression change data determination module is also used to: determine the quantity difference between the current quantity and the historical quantity; if the quantity difference is greater than zero, determine that the virtual object has a first type of expression change process; obtain the expression change data corresponding to the first type of expression change process from a preset expression database; if the quantity difference is less than zero, determine that the virtual object has a second type of expression change process; wherein the first type of expression change process and the first type of expression change process are used to represent expression change processes corresponding to two different emotions; obtain the expression change data corresponding to the second type of expression change process from the preset expression database.

[0016] In some embodiments, the device also includes: a position identification module, which is used to identify the position of the target object based on the environmental video to obtain the position change information of the target object; an analysis module, which is used to determine that the virtual object has a third type of expression change process if the position change information of the target object meets a preset position change condition; the expression change data determination module is also used to obtain expression change data corresponding to the third type of expression change process from a preset expression database; the data adding module is also used to add the expression change data corresponding to the third type of expression change process to the first rendering data.

[0017] In some embodiments, the determination module is also used to: perform virtual object identification on the current playback screen to obtain the virtual object in the current playback screen; if the action type of the target object is a preset type, determine an interactive action matching the action type from a preset action library; obtain second rendering data of the virtual object performing the interactive action; and use the second rendering data to perform image rendering to obtain the interactive video.

[0018] In some embodiments, the device further includes: a detection module for detecting whether a new interactive video is generated after playing the interactive video; and a video playback module for continuing to play the pre-produced video if no new interactive video exists.

[0019] An embodiment of the present application provides an electronic device, comprising: a memory for storing executable instructions; and a processor for implementing the above-mentioned image processing method when executing the executable instructions stored in the memory.

[0020] An embodiment of the present application provides a computer program product, which includes executable instructions stored in a computer-readable storage medium; wherein a processor of an electronic device reads the executable instructions from the computer-readable storage medium and implements the above-mentioned image processing method when executing the executable instructions.

[0021] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute the executable instructions to implement the above-mentioned image processing method.

[0022] The embodiments of the present application have the following beneficial effects: in the process of displaying a prefabricated video on a display device, an environmental video located within a preset range of the display device is collected by an image acquisition device of the display device; if it is determined based on the environmental video that a target object exists within the preset range and the current state of the target object meets the preset state index, the current playback screen of the prefabricated video is obtained; based on the current playback screen, an interactive video matching the current state of the target object is determined, and the interactive video is displayed on the display device. In this way, by providing interaction between the target object and the virtual object in the prefabricated video during the process of displaying the prefabricated video on the display device, an interactive video is generated, and the interaction between the target object and the prefabricated video can be realized without resources from other channels, realizing the expansion of in-depth and efficient interactive methods in a resource-intensive manner during the process of playing the prefabricated video on the display device, thereby promoting the playback quality of the prefabricated video. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is an optional architecture diagram of an image processing system provided in an embodiment of the present application;

[0024] Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0025] Figure 3 It is an optional flowchart of the image processing method provided in the embodiment of the present application;

[0026] Figure 4 It is a schematic diagram of the implementation process of obtaining the current playback screen of a pre-made video provided in an embodiment of the present application;

[0027] Figure 5 It is a schematic diagram of an implementation process of determining an interactive video provided in an embodiment of the present application;

[0028] Figure 6 is another optional flowchart of the image processing method provided in the embodiment of the present application;

[0029] Figure 7 It is a schematic diagram of an implementation flow of determining expression change data of a virtual object provided in an embodiment of the present application;

[0030] Figure 8 This is another optional flowchart of the image processing method provided in the embodiment of the present application;

[0031] Fig. 9 is a flowchart of an image processing method jointly performed by a 3D display device and a server provided in an embodiment of the present application;

[0032] Fig.10 It is a schematic diagram of the display effect of the naked-eye 3D screen provided in an embodiment of the present application;

[0033] Fig.11 It is a schematic diagram of the display effect of automatically playing pre-made content provided in an embodiment of the present application;

[0034] Fig.12 It is a schematic diagram of the display effect of executing the greeting action provided in an embodiment of the present application;

[0035] Fig.13 It is a schematic diagram of the display effect of generating special effects provided in an embodiment of the present application;

[0036] Fig.14 It is an overall flow chart of the image processing method provided in the embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0038] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meaning as those commonly understood by those skilled in the art of the technical field of the embodiments of the present application. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0039] Before explaining the image brightness processing method of the embodiment of the present application, the professional terms involved in the embodiment of the present application are first explained.

[0040] (1) Naked-eye 3D screen: refers to a display system that uses the parallax characteristics of the human eyes to obtain realistic three-dimensional images with space and depth without the need for any auxiliary equipment (such as 3D glasses, helmets, etc.).

[0041] (2) Automatic identification technology: It is a technology that uses a certain identification device to automatically obtain relevant information of the identified object through the proximity between the identified object and the identification device, and provides it to the background computer processing system to complete the relevant subsequent processing. Automatic identification technology integrates computer, optical, electrical, communication and network technologies, and combines with the Internet, mobile communication and other technologies to achieve the positioning of objects and information sharing on a global scale, thereby giving objects intelligence and realizing communication and dialogue between people and objects and between objects.

[0042] (3) Camera: used to capture the audience's movements and postures and transmit the data to the processor for processing.

[0043] (4) Processor: refers to the computing and control core of a computer system and is the final execution unit for information processing and program execution.

[0044] (5) Game engine: refers to some pre-programmed editable computer game systems or core components of some interactive real-time graphics applications. These systems provide game designers with various tools needed to write games. The purpose is to allow game designers to easily and quickly create game programs without having to start from scratch.

[0045] The method in the related art is described below.

[0046] In the related technology, the naked-eye 3D screen is only used as a pure content material display without interactive experience. This display method is too simple: because there is no naked-eye 3D screen to interact with users, it is basically in a pure display state, and it is difficult for users to connect with its content. In addition, the user's sense of participation is not enough: because there is no story packaging and no interactive plot, it is difficult for users to have an emotional connection with the displayed content in this process, thereby increasing their memory and consumption of the content.

[0047] In order to solve at least one of the above-mentioned problems existing in the related art, an embodiment of the present application provides an image processing method, which adds a recognition device to the naked-eye 3D screen to enable the user to interact with the screen; at the same time, it enhances the user's sense of participation, and can set interactive actions according to the displayed content, which can be a gesture, an action, etc., or even the change of screen content can be triggered when a certain number of people gather, giving the user an experience of opening an Easter egg, allowing the user to actively participate, and have a greater sense of immersion and experience.

[0048] The image processing method provided by the embodiment of the present application firstly collects the environment video located within the preset range of the display device through the image acquisition device of the display device during the process of displaying the prefabricated video on the display device; then, in response to determining that the target object exists within the preset range based on the environment video and the current state of the target object meets the preset state index, the current playback screen of the prefabricated video is obtained; finally, based on the current playback screen, an interactive video matching the current state of the target object is determined, and the interactive video is displayed on the display device. In this way, by providing the interaction between the target object and the virtual object in the prefabricated video during the process of displaying the prefabricated video on the display device, an interactive video is generated, and the interaction between the target object and the prefabricated video can be realized without resources from other channels, realizing the expansion of in-depth and efficient interaction methods in the process of playing the prefabricated video on the display device in a resource-intensive manner, thereby promoting the playback quality of the prefabricated video.

[0049] Here, firstly, an exemplary application of the image processing device of the embodiment of the present application is described, and the image processing device is an electronic device for implementing the image processing method. In one implementation, the image processing device (i.e., electronic device) provided in the embodiment of the present application can be implemented as a terminal. In the process of implementation, the terminal can be a display device, and the display device can be a 3D display device, for example, the 3D display device can be a naked eye 3D screen; in another implementation, the image processing device provided in the embodiment of the present application can also be implemented as a server, wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN, Content Delivery Network), and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected by wired or wireless communication, which is not limited in the embodiment of the present application. Below, an exemplary application of the image processing device when it is implemented as a 3D display device will be described.

[0050] See also Figure 1 , Figure 1 This is an optional architecture diagram of the image processing system provided by the embodiment of the present application. The embodiment of the present application is described by taking the application of the image processing method to any video application, and the video application is run on a 3D display device as an example. In the process of running the video application on the 3D display device, the 3D display device can display pre-made videos, and can generate interactive videos based on the number of viewers within the preset range of the 3D display device and the actions of the viewers, so as to realize the interaction between the pre-made videos in the 3D display device and the viewers, realize the immersive viewing experience of the viewers, and improve the viewing experience of the users.

[0051] In the embodiment of the present application, the image processing system 10 includes at least a terminal (i.e., a 3D display device 100), a network 200, and a server 300. The server 300 may be a server for a video application. The 3D display device 100 may constitute an image processing device in the embodiment of the present application. The 3D display device 100 is connected to the server 300 via the network 200, and the network 200 may be a wide area network or a local area network, or a combination of the two.

[0052] In the embodiment of the present application, a video application is installed on the 3D display device 100. The producer of the pre-made video can play the pre-made video through the video application on the 3D display device 100. In the process of playing the pre-made video, the image acquisition device of the 3D display device collects the environment video located in the preset range of the 3D display device; and sends the environment video to the server 300 through the network 200. The server 300 identifies and analyzes the environment video, thereby determining whether there is a target object within the preset range based on the environment video, and determining whether the current state of the target object meets the preset state index, and sends the analysis result obtained after the analysis to the terminal 100. If the terminal determines that the target object exists within the preset range and the current state of the target object meets the preset state index based on the analysis result, the current playback screen of the pre-made video is obtained; based on the current playback screen, an interactive video matching the current state of the target object is determined, and the interactive video is displayed on the 3D display device 100.

[0053] The above embodiment describes a scheme for implementing an image processing method by interaction between a terminal and a server. In another implementation mode, a video application is installed on the 3D display device 100. The producer of the pre-made video can play the pre-made video through the video application on the 3D display device 100. In the process of playing the pre-made video, the image acquisition device of the 3D display device collects the environment video located in the preset range of the 3D display device; and sends the environment video to the server 300 through the network 200. The server 300 identifies and analyzes the environment video, thereby determining whether there is a target object within the preset range based on the environment video, and determining whether the current state of the target object meets the preset state index, and sends the analysis result obtained after the analysis to the terminal 100. If the terminal determines that there is a target object within the preset range and the current state of the target object meets the preset state index based on the analysis result, the current playback screen of the pre-made video is obtained, and the current playback screen is sent to the server 300. The server 300 determines the interactive video matching the current state of the target object based on the current playback screen, and sends the interactive video to the 3D display device 100, and displays the interactive video on the 3D display device 100.

[0054] In another implementation, the image processing method may be independently completed by the 3D display device 100, that is, in the process of displaying the pre-produced video on the 3D display device, the image acquisition device of the 3D display device 100 acquires an environmental video located within a preset range of the 3D display device 100; the 3D display device 100 determines whether there is a target object within the preset range based on the environmental video, and determines whether the current state of the target object satisfies a preset state indicator; if it is determined that the target object exists within the preset range and the current state of the target object satisfies the preset state indicator, the current playback screen of the pre-produced video is acquired; the 3D display device 100 determines an interactive video matching the current state of the target object based on the current playback screen, and displays the interactive video on the 3D display device.

[0055] The image processing method provided in the embodiment of the present application can also be implemented based on a cloud platform and through cloud technology. For example, the server 300 can be a cloud server. The cloud server determines whether there is a target object within a preset range based on the environmental video, and determines whether the current state of the target object meets the preset state index, or determines an interactive video that matches the current state of the target object based on the current playback screen.

[0056] In some embodiments, a cloud storage may be provided, and the pre-made video or interactive video may be stored in the cloud storage. Thus, when the 3D display device is powered on and receives a playback request from the producer of the pre-made video, the pre-made video may be obtained from the cloud storage for playback.

[0057] It should be noted here that cloud technology refers to a hosting technology that unifies hardware, software, network and other resources in a wide area network or local area network to achieve data computing, storage, processing and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool that is used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, in the future, each item may have its own identification mark, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately. All kinds of industry data require strong system backing support, which can be achieved through cloud computing.

[0058] Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, Figure 2The electronic device shown may be an image processing device, which includes: at least one processor 310, a memory 350, at least one network interface 320, and a user interface 330. The various components in the image processing device are coupled together via a bus system 340. It is understood that the bus system 340 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 340 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 340 is not described in detail. Figure 2 Various buses are labeled as bus system 340 .

[0059] The processor 310 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0060] The user interface 330 includes one or more output devices 331 that enable presentation of media content, and one or more input devices 332 .

[0061] The memory 350 may be removable, non-removable or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drive, optical disk drive, etc. The memory 350 may optionally include one or more storage devices physically away from the processor 310. The memory 350 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 350 described in the embodiment of the present application is intended to include any suitable type of memory. In some embodiments, the memory 350 can store data to support various operations, and examples of these data include programs, modules, and data structures or subsets or supersets thereof, as exemplarily described below.

[0062] An operating system 351 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic businesses and processing hardware-based tasks; a network communication module 352 is used to reach other computing devices via one or more (wired or wireless) network interfaces 320. Exemplary network interfaces 320 include: Bluetooth, wireless compatibility certification (WiFi), and Universal Serial Bus (USB), etc.; an input processing module 353 is used to detect one or more user inputs or interactions from one of the one or more input devices 332 and translate the detected inputs or interactions.

[0063] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 An image processing device 354 stored in the memory 350 is shown. The image processing device 354 may be an image processing device in an electronic device. The image processing device 354 may be software in the form of a program or a plug-in, and includes the following software modules: an acquisition model 3541, an acquisition module 3542, and a determination module 3543. These modules are logical, and therefore may be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0064] In some embodiments, the device provided in the embodiments of the present application can be implemented in hardware. As an example, the device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the image processing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), or other electronic components.

[0065] The image processing methods provided in the embodiments of the present application may be executed by an electronic device, wherein the electronic device may be a server or a display device, for example, a 3D display device, that is, the image processing methods in the embodiments of the present application may be executed by a server, or by a 3D display device, or may be executed through interaction between a server and a 3D display device.

[0066] Figure 3This is an optional flow chart of the image processing method provided in the embodiment of the present application. Figure 3 The steps shown are explained, Figure 3 In the above, the execution subject of the image processing method is a 3D display device as an example for explanation. In the embodiment of the present application, the 3D display device includes a naked-eye 3D screen, and may also have an image acquisition device and a processor. The naked-eye 3D screen is used to implement the display steps of the video display in the embodiment of the present application. The logical steps of data processing and calculation by the 3D display device in the embodiment of the present application can be implemented by the processor in the 3D display device. Figure 3 The method includes the following steps S101 to S103:

[0067] Step S101 , when a 3D display device displays a pre-made video, an image acquisition device of the 3D display device acquires an environment video within a preset range of the 3D display device.

[0068] Here, the 3D display device may be a naked-eye 3D screen realized by using autostereoscopy technology. Naked-eye 3D technology refers to the general term for technologies that achieve stereoscopic visual effects without the help of external tools such as polarized glasses. Naked-eye 3D technology mainly includes light barrier technology, slit liquid crystal grating technology, cylindrical lens technology, pointing light source technology, active backlight, etc. Among them, the principle of slit liquid crystal grating technology is that after adding a slit grating in front of the screen of the 3D display device, when the image that should be seen by the left eye is displayed on the liquid crystal screen, the opaque stripes will block the right eye; similarly, when the image that should be seen by the right eye is displayed on the liquid crystal screen, the opaque stripes will block the left eye, and the viewer can see the 3D image by separating the visible images of the left eye and the right eye. The principle of cylindrical lens technology is to project the pixels corresponding to the left and right eyes respectively into the left and right eyes through the refraction principle of the lens to achieve image separation. Compared with the slit grating technology, the biggest advantage is that the lens will not block the light, so the brightness has been greatly improved. Pointing light source technology is to precisely control two sets of screens to project images to the left and right eyes respectively.

[0069] In the embodiment of the present application, the 3D display device can use any naked-eye 3D technology to achieve the naked-eye 3D effect.

[0070] In the embodiment of the present application, the 3D display device can be arranged outdoors as a marketing device, and the producer of the pre-made video can request to display the pre-made video on the 3D display device, and the pre-made video can be an advertising video. During the playback of the pre-made video, the viewer can move to a preset range of the 3D display device to watch the pre-made video with naked eye 3D effect.

[0071] In some embodiments, the 3D display device may be provided with an image acquisition device, which is used to acquire the environmental video within the preset range of the 3D display device in real time during the process of the 3D display device displaying the pre-made video. The preset range of the 3D display device may refer to the area within the preset diameter distance in front of the 3D display device.

[0072] In the process of collecting environmental videos, a segment of environmental video can be collected at a preset time interval, and image recognition is performed on the video frames in each segment of the environmental video, thereby realizing analysis of the environmental video.

[0073] In some embodiments, the 3D display device may include a processor. After capturing the ambient video, the 3D display device may analyze the ambient video using its own processor. In other embodiments, after capturing the ambient video, the 3D display device may send the ambient video to a server, which may analyze the ambient video.

[0074] Step S102, in response to determining that a target object exists within a preset range based on the environmental video and that a current state of the target object meets a preset state indicator, a current playback screen of the pre-produced video is obtained.

[0075] In an embodiment of the present application, image recognition can be performed on video frames of an environmental video captured at preset time intervals to identify whether there is a viewer (i.e., a target object) in the video frame, and to analyze the current state of the target object. Alternatively, changes in the number of viewers in two previous and subsequent environmental videos (including but not limited to changes in quantity and current state) can be analyzed.

[0076] Here, character recognition technology can be used to identify each video frame to determine whether there is a viewer in the video frame. If there is at least one viewer in any video frame, it can be determined that there is a viewer in the environmental video where the video frame is located, that is, based on the environmental video, it is determined that there is a target object within a preset range. When it is determined that there is a target object in the environmental video, the video frame can be further identified to determine the current state of the target object. Among them, the current state of the target object includes but is not limited to: the number of target objects, the action type of the target object, etc. It should be noted that when the current state of the target object is the number of target objects, there are one or more target objects in the environmental video, and when the current state of the target object is the action type of the target object, the number of target objects in the environmental video can be a preset number, and the preset number can be one or more, and the preset number is related to the action type that the processor can recognize.

[0077] In some embodiments, when the current state of the target object is the number of target objects, the preset state indicator may be an indicator related to the number of target objects, and whether the current state of the target object satisfies the preset state indicator may be determined based on the number of target objects. Figure 4 , Figure 4 It is shown that step S102 can be implemented by the following steps S1021 to S1023:

[0078] Step S1021, performing video recognition on the environment video to obtain a video recognition result for a preset range.

[0079] In one implementation, video recognition is performed on an environmental video to obtain a video recognition result for a preset range in the following manner: first, video frames are captured on the environmental video to obtain multiple video frames; then, each video frame is subjected to a content consistency analysis with an image in a preset image library to obtain a consistency analysis result; if the consistency analysis result between the video frame and any image in the preset image library is consistent, attribute information corresponding to the image is obtained from the preset image library; then, a video frame recognition result corresponding to the video frame is determined based on the attribute information; finally, all video frame recognition results in the environmental video are summarized to obtain a video recognition result for the environmental video.

[0080] In some embodiments, a plurality of images are stored in a preset image library, each image having a tag information, the tag information constituting the attribute information of the image, and the tag information being used to identify the number of target objects in the corresponding image. In the process of video recognition, each video frame in the environmental video can be compared with each image in the preset image library, and the consistency between the video frame and each image can be analyzed to obtain a consistency analysis result. If it is determined that the video frame is consistent with a certain image, the video frame recognition result corresponding to the video frame can be determined based on the attribute information. In the implementation process, the tag information of the image can be obtained, and the number of target objects identified in the tag information can be determined as the number of target objects in the corresponding video frame, and the number of target objects constitutes the video frame recognition result of the video frame.

[0081] Summarizing the video results of all video frames in the environment video can determine whether the number of target objects in multiple video frames in the environment video is the same. If the number of target objects in multiple video frames in the environment video is the same, the number of the same target objects is determined as the final video recognition result of the environment video.

[0082] If the number of target objects in multiple video frames in the environmental video is different, the number of target objects in each video frame can be obtained, the number of target objects can be counted, and the number with the most occurrences can be determined as the final video recognition result of the environmental video; for example, if the environmental video is divided into 10 video frames, where the video frame recognition results after identifying the number of target objects in the 10 video frames are: 2, 2, 3, 4, 4, 4, 5, 4, 0, 0, then it can be seen that the number of target objects is 4 and appears the most times, therefore, it can be determined that the final video recognition result of the environmental video is that the number of target objects is 4. Alternatively, the largest number can also be determined as the final video recognition result of the environmental video; for example, for the aforementioned video frame recognition results, where the largest number of target objects is 5, therefore, it can be determined that the final video recognition result of the environmental video is that the number of target objects is 5. Alternatively, the number of consecutive occurrences of the same number of times is the largest can also be determined as the final video recognition result of the environmental video; for example, for the aforementioned video frame recognition result, the number of consecutive occurrences of 2 is twice, the number of consecutive occurrences of 4 is three times, and the number of consecutive occurrences of 0 is two times. Therefore, it can be determined that the final video recognition result of the environmental video is that the number of target objects is 4. Of course, the embodiments of the present application are not limited to the above-mentioned methods for determining the final video recognition results. Other methods for determining the final video recognition results also fall within the protection scope of the embodiments of the present application.

[0083] In some embodiments, a plurality of images may be stored in a preset image library, each image having a tag information, the tag information constituting the attribute information of the image, and the tag information being used to identify the action type of the target object in the corresponding image. In the process of video recognition, each video frame in the environmental video may be compared with each image in the preset image library, and the consistency between the video frame and each image may be analyzed to obtain a consistency analysis result. If it is determined that the video frame is consistent with a certain image, the video frame recognition result corresponding to the video frame may be determined based on the attribute information. In the implementation process, the tag information of the image may be obtained, and the action type of the target object identified in the tag information may be determined as the action type of the target object in the corresponding video frame, and the action type of the target object constitutes the video frame recognition result of the video frame.

[0084] Summarizing the video results of all video frames in the environmental video can be to connect the action types of the target objects in multiple video frames in the environmental video to obtain the action type corresponding to the entire environmental video, and finally, determine the action type corresponding to the entire environmental video as the final video recognition result of the environmental video.

[0085] In another implementation, performing video recognition on environmental videos and obtaining video recognition results for a preset range can be achieved in the following manner: first, inputting the environmental video into a pre-trained video object quantity recognition model, and identifying the number of target objects in the environmental video through the video object quantity recognition model to obtain the number of target objects for the preset range; finally, inputting the environmental video into a pre-trained video action recognition model, and identifying the actions of the target objects in the environmental video through the video action recognition model to obtain the action types of the target objects for the preset range.

[0086] In an embodiment of the present application, a video object quantity recognition model can be pre-trained by artificial intelligence technology, and the video object quantity recognition model can accurately recognize the number of target objects in the environmental video. The video object quantity recognition model is a model obtained by training based on a large amount of sample data that is manually annotated in advance. By inputting the environmental video into the pre-trained video object quantity recognition model, and performing a series of calculations on the environmental video using the pre-trained model parameters in the video object quantity recognition model, the number of target objects in the environmental video can be outputted in the end, that is, the number of target objects for a preset range is obtained.

[0087] It is also possible to pre-train a video action recognition model through artificial intelligence technology, and the video action recognition model can accurately identify the action type of the target object in the environmental video. The video action recognition model is a model trained based on a large amount of sample data that has been manually annotated in advance. By inputting the environmental video into the pre-trained video action recognition model, and performing a series of calculations on the environmental video through the pre-trained model parameters in the video action recognition model, the action type of the target object in the environmental video can be output, that is, the number of target objects for a preset range can be obtained.

[0088] Step S1022: If the video recognition result indicates that there are target objects within the preset range, the number of target objects is determined; the number of target objects constitutes the current state of the target objects.

[0089] Step S1023: If the number of target objects is greater than the number threshold, determine whether the current state of the target object meets the preset state indicator, and obtain the current playback screen of the pre-produced video.

[0090] In an embodiment of the present application, it can be determined that the number of target objects is greater than a number threshold. If the number of target objects is greater than the number threshold, it can be determined that the number of target objects currently within the preset range of the 3D display device is large, indicating that the current state of the target object meets the preset state indicator. When the current state of the target object meets the preset state indicator, an interactive video for interacting with the target object can be further generated. The interactive video is formed based on the screen in the pre-produced video currently being played. This ensures that the pre-produced video being played and the interactive video can be connected, thereby improving the viewing experience of the target object. Therefore, it is necessary to obtain the current playback screen of the pre-produced video.

[0091] In some other embodiments, the preset state indicator may be an indicator related to the action type of the target object, and whether the current state of the target object meets the preset state indicator may be determined based on the action type of the target object. Figure 4 , Figure 4 It is also shown that step S102 can also be implemented by steps S1021, S1024 and S1025:

[0092] Step S1024: if the video recognition result indicates that there is a target object within the preset range, determine the action type of the target object; the action type of the target object constitutes the current state of the target object.

[0093] Step S1025: If the action type of the target object is a preset type, determine whether the current state of the target object meets the preset state index, and obtain the current playback screen of the pre-produced video.

[0094] Here, there may be a preset type library, in which a plurality of preset types are stored. After the action type of the target object is determined, a match may be made from the preset type library to determine whether there is a preset type that is the same as the action type in the preset type library. For example, the preset type library may store a preset type of high-five. If it is determined that the action type of the target object in the environment video is high-five, it may be determined that the current state of the target object meets the preset state indicator, and thus the current playback screen of the pre-made video may be obtained.

[0095] Step S103: Based on the current playing picture, determine an interactive video that matches the current state of the target object, and display the interactive video on a 3D display device.

[0096] In an embodiment of the present application, the preset type library may also store interactive action types corresponding to each preset type. When it is determined that the current state of the target object meets the preset state indicator, the interactive action type corresponding to the corresponding preset type may be simultaneously obtained from the preset type library. The interactive action type is the action type that needs to be applied to the virtual object in the pre-produced video.

[0097] During the implementation process, the virtual object in the current playback screen can be identified based on the current playback screen, and the interactive action corresponding to the interactive action type can be applied to the virtual object, and the virtual object can be controlled to perform the interactive action, thereby generating an interactive video that matches the current state of the target object.

[0098] In some embodiments, if the interactive video matches the current state of the target object, see Figure 5 , Figure 5 It is shown that determining the interactive video in step S103 can be achieved by following steps S1031 to S1034:

[0099] Step S1031, performing virtual object recognition on the current playing picture to obtain the virtual object in the current playing picture.

[0100] Step S1032: If the number of target objects is greater than the number threshold, determine an interactive action that matches the number of target objects for the virtual object.

[0101] Step S1033: obtaining first rendering data of the virtual object performing an interactive action.

[0102] Step S1034: Use the first rendering data to perform image rendering to obtain an interactive video.

[0103] The embodiment of the present application corresponds to the following scenario: when it is detected that the number of target objects within the preset range of the 3D display device is greater than the number threshold, the change of the content displayed by the 3D display device can also be triggered. For example, when it is detected that the number of viewers within the preset range of the 3D display device is greater than the number threshold, the displayed virtual object can be controlled to make a welcome gesture and show a happy expression.

[0104] In other embodiments, the engine in the video application can be used to render interactive videos in real time, so that the target object can control the actions of the virtual object through its own actions and gestures. Figure 5 , Figure 5 It is also shown that step S103 can also be implemented through steps S1031, S1035 to S1037:

[0105] Step S1035: if the action type of the target object is a preset type, determine an interactive action matching the action type from a preset action library.

[0106] Step S1036, obtaining second rendering data of the virtual object performing the interactive action.

[0107] Step S1037: Use the second rendering data to perform image rendering to obtain an interactive video.

[0108] The embodiment of the present application corresponds to the following scenario: when the target object within the preset range of the 3D display device is detected to have a preset action type, the change of the content displayed by the 3D display device can also be triggered. For example, when the viewer within the preset range of the 3D display device is detected to make a high-five action, the displayed virtual object can be controlled to respond to the high-five.

[0109] The image processing method provided by the embodiment of the present application, in the process of displaying the pre-made video on the 3D display device, collects the environment video located in the preset range of the 3D display device through the image acquisition device of the 3D display device; if it is determined based on the environment video that there is a target object in the preset range, and the current state of the target object meets the preset state index, obtain the current playback screen of the pre-made video; based on the current playback screen, determine the interactive video that matches the current state of the target object, and display the interactive video on the 3D display device. In this way, by providing the interaction between the target object and the virtual object in the pre-made video during the display of the pre-made video on the 3D display device, an interactive video is generated, and the interaction between the target object and the pre-made video can be realized without resources from other channels, realizing the expansion of in-depth and efficient interactive methods in the process of playing the pre-made video on the 3D display device in a resource-intensive manner, and promoting the playback quality of the pre-made video.

[0110] In some embodiments, the environmental video has a preset duration, and a segment of the environmental video is captured every preset duration; the change in the number of target objects within a preset range within a certain period of time can also be determined, so that the virtual objects in the prefabricated video are controlled to have different expressions according to the change in the number of target objects within a certain period of time. Figure 6 is another optional flow chart of the image processing method provided in the embodiment of the present application. In the implementation process, see Figure 6 ,exist Figure 5 On the basis of obtaining the first rendering data, the method may further include the following steps S201 to S205:

[0111] Step S201, obtaining a current environment video within a current preset duration and a historical environment video within a historical preset duration that is connected to the current environment video and is located before the current preset duration.

[0112] Step S202, performing video recognition on the current environment video and the historical environment video respectively, and obtaining a current video recognition result and a historical video recognition result accordingly.

[0113] Here, the current video recognition result includes but is not limited to the number of target objects and the action type of the target objects in the current environment video; the historical video recognition result includes but is not limited to the number of target objects and the action type of the target objects in the historical environment video.

[0114] Step S203, extracting the number of target objects from the current video recognition result and the historical video recognition result respectively, and obtaining the current number and the historical number of target objects accordingly.

[0115] Step S204, determining expression change data of the virtual object based on the current number and the historical number.

[0116] In some embodiments, see Figure 7 , Figure 7 It is shown that step S204 determining the expression change data of the virtual object can be implemented by following steps S2041 to S2045:

[0117] Step S2041, determining the quantity difference between the current quantity and the historical quantity.

[0118] Step S2042: If the quantity difference is greater than zero, it is determined that the virtual object has a first type of expression change process.

[0119] In an embodiment of the present application, the first type of expression change process can be a positive expression change process. Since the quantity difference represents the quantity difference between the current quantity and the historical quantity, if the quantity difference between the current quantity and the historical quantity is greater than zero, it indicates that the number of viewers within the preset range of the 3D display device is increasing. Therefore, the virtual object can have a positive first type of expression change process. For example, the first type of expression change process can be a happy expression change process.

[0120] Step S2043, acquiring expression change data corresponding to the first type of expression change process from a preset expression database.

[0121] Here, the preset expression database stores expression change data corresponding to different types of expression change processes, and the face of the virtual object is rendered based on the expression change data to present the corresponding expression change process of the virtual object. When acquiring the expression change data, the corresponding expression change data can be queried from the preset expression database using the first type of expression change process as a key.

[0122] Step S2044, if the quantity difference is less than zero, it is determined that the virtual object has a second type of expression change process; wherein the first type of expression change process and the first type of expression change process are used to represent expression change processes corresponding to two different emotions.

[0123] The second type of expression change process can be a reverse expression change process. Since the quantity difference represents the quantity difference between the current quantity and the historical quantity, if the quantity difference between the current quantity and the historical quantity is less than zero, it indicates that the number of viewers within the preset range of the 3D display device is decreasing. Therefore, the virtual object can have a reverse second type of expression change process. For example, the second type of expression change process can be a sad expression change process.

[0124] Step S2045, acquiring expression change data corresponding to the second type of expression change process from a preset expression database.

[0125] Here, the preset expression database stores expression change data corresponding to different types of expression change processes, and the face of the virtual object is rendered based on the expression change data to present the corresponding expression change process of the virtual object. When acquiring the expression change data, the corresponding expression change data can be queried from the preset expression database using the second type of expression change process as a key.

[0126] Step S205: adding the expression change data to the first rendering data.

[0127] In some embodiments, the decision-making behavior of the target object can also be determined. If the decision-making behavior of the target object is consistent with the sales response behavior of the producer of the pre-produced video, the virtual object in the pre-produced video can be made to respond with a positive expression. If the decision-making behavior of the target object is inconsistent with the sales response behavior of the producer of the pre-produced video, the virtual object in the pre-produced video can be made to respond with a reverse expression.

[0128] Figure 8 This is another optional flow chart of the image processing method provided in the embodiment of the present application. In the process of implementation, see Figure 8 ,exist Figure 5 On the basis of obtaining the first rendering data, the method may further include the following steps S301 to S304:

[0129] Step S301: Based on the environment video, the position of the target object is identified to obtain the position change information of the target object.

[0130] Here, the environment video can be identified to determine the position information of the target object in each video frame of the environment video, and the position change information of the target object can be obtained by analyzing the position information of the target object in multiple consecutive video frames. The position change information can be the position change trajectory of the target object. After the position change trajectory is obtained, the trajectory end point of the position change trajectory can be determined.

[0131] Step S302: If the position change information of the target object meets a preset position change condition, it is determined that the virtual object has a third type of expression change process.

[0132] In some embodiments, it can be determined whether the position change trajectory of the target object coincides with the preset position change trajectory, or it can be determined whether the trajectory end point of the target object coincides with the preset end point position. If they coincide, it can be determined that the target object has performed a positive feedback action, and therefore, it can be determined that the virtual object has a third type of expression change process. The third type of expression change process can be a positive expression change process. For example, the positive expression change process can be a happy expression change process.

[0133] In other embodiments, if the position change trajectory of the target object does not overlap with the preset position change trajectory, or the target object's trajectory end point does not overlap with the preset end point position, it can be determined that the target object has performed a reverse feedback action, and therefore, it can be determined that the virtual object has a fourth type of expression change process. The fourth type of expression change process can be a reverse expression change process. For example, the reverse expression change process can be a regretful or disappointed expression change process.

[0134] Step S303, acquiring expression change data corresponding to the third type of expression change process from a preset expression database.

[0135] Step S304: adding the expression change data corresponding to the third type of expression change process to the first rendering data.

[0136] The embodiments of the present application can be applied to the following scenarios: if the 3D display device is a display screen of an e-sports hotel peripheral, the 3D display device can display the promotional advertisement of the e-sports hotel. After the viewer interacts with the virtual person in the promotional advertisement, the 3D display device can also continuously obtain the viewer's position change information to determine whether the viewer enters the hotel to check in, that is, to determine whether the viewer converts the played promotional advertisement. If the viewer enters the hotel to check in, the virtual person in the promotional advertisement can be controlled to show a happy expression. If the viewer does not enter the hotel to check in, the virtual person in the promotional advertisement can be controlled to show a regretful or disappointed expression.

[0137] In some embodiments, after playing the interactive video, it is also possible to detect whether a new interactive video is generated; if there is no new interactive video, the pre-made video continues to be played.

[0138] In some embodiments, the image processing system includes at least a 3D display device and a server. The 3D display device can be a naked-eye 3D screen. A video application can be run on the 3D display device, and a pre-produced video can be played through the video application. The server constitutes a background server of the video application. A series of operations can be performed through the server to generate an interactive video for real-time interaction with viewers of the pre-produced video, thereby achieving the ability to play an interactive video for real-time interaction with viewers during the playback of the pre-produced video.

[0139] In the embodiment of the present application, the 3D display device includes a naked eye 3D screen and may also have an image acquisition device. The naked eye 3D screen is used to implement the display step of the video display in the embodiment of the present application, and the logical steps of data processing and calculation of the 3D display device in the embodiment of the present application can all be executed by the server.

[0140] Based on the above-mentioned image processing method performed by the 3D display device, an embodiment of the present application further provides an image processing method, which can be implemented by the 3D display device and the server together. Fig. 9 is a flow chart of an image processing method jointly performed by a 3D display device and a server provided in an embodiment of the present application, such as Fig. 9 As shown, the method includes the following steps S401 to S415:

[0141] Step S401: The 3D display device receives a video playback operation from a producer of a pre-made video.

[0142] Here, the video playing operation may be a triggering operation performed on a client of a video application, and the video playing operation is used to trigger the playing of a pre-produced video.

[0143] Step S402: The 3D display device generates a video play request in response to the video play operation.

[0144] The video play request is used to request the server to play the pre-made video on the 3D display device. The video play request includes the video identifier of the pre-made video.

[0145] Step S403: The 3D display device sends a video playback request to the server.

[0146] Step S404: In response to the video play request, the server obtains a pre-produced video corresponding to the video identifier from a pre-produced video library.

[0147] Here, at least one prefabricated video is stored in the prefabricated video library, and the corresponding prefabricated video can be obtained based on the video identifier in the video playback request.

[0148] Step S405: The server sends the pre-made video to the 3D display device.

[0149] Step S406: The 3D display device displays the pre-produced video.

[0150] Step S407 , when the 3D display device displays the pre-made video, an environment video within a preset range of the 3D display device is captured by an image capture device of the 3D display device.

[0151] The 3D display device may be provided with an image acquisition device, which is used to acquire the environmental video within the preset range of the 3D display device in real time during the process of the 3D display device displaying the pre-made video. The preset range of the 3D display device may refer to the area within the preset diameter distance in front of the 3D display device. In the process of acquiring the environmental video, a segment of the environmental video may be acquired at a preset time interval, and the image recognition of the video frames in each segment of the environmental video may be performed to realize the analysis of the environmental video.

[0152] Step S408: The 3D display device sends the environment video to the server.

[0153] Step S409: The server determines whether there is a target object within a preset range based on the environmental video, and determines whether the current state of the target object meets a preset state indicator to obtain an analysis result.

[0154] Here, the 3D display device can perform image recognition on the video frames of the ambient video captured at preset time intervals, identify whether there is a viewer (i.e., the target object) in the video frame, and analyze the current state of the target object. Alternatively, it can also analyze the changes in the viewers in the two ambient videos sent before and after (including but not limited to changes in quantity and changes in current state).

[0155] A person recognition technology can be used to identify each video frame to determine whether there is a viewer in the video frame. If there is at least one viewer in any video frame, it can be determined that there is a viewer in the environment video where the video frame is located, that is, it is determined based on the environment video that the target object exists within a preset range. When it is determined that there is a target object in the environment video, the video frame can be further identified to determine the current state of the target object. The current state of the target object includes but is not limited to: the number of target objects, the action type of the target object, etc.

[0156] It should be noted that, for other implementation methods of determining whether there is a target object within a preset range based on the ambient video, and determining whether the current state of the target object meets the preset state indicators, reference can be made to the content of step S102 in the above embodiment. The only difference is that in step S102, the 3D display device determines whether there is a target object within a preset range based on the ambient video, and determines whether the current state of the target object meets the preset state indicators, while in this step, the server determines whether there is a target object within a preset range based on the ambient video, and determines whether the current state of the target object meets the preset state indicators.

[0157] Step S410: The server sends the analysis result to the 3D display device.

[0158] Step S411: if it is determined based on the analysis result that a target object exists within a preset range and the current state of the target object meets a preset state indicator, the 3D display device obtains a current playback screen of the pre-produced video.

[0159] Step S412: The 3D display device sends the current playing picture to the server.

[0160] Step S413: The server determines an interactive video that matches the current state of the target object based on the current playback screen.

[0161] It should be noted that for other implementation methods of determining an interactive video that matches the current state of the target object based on the current playback screen, please refer to the content of step S103 in the above embodiment. The only difference is that in step S103, the 3D display device determines the interactive video that matches the current state of the target object based on the current playback screen, while in this step, the server determines the interactive video that matches the current state of the target object based on the current playback screen.

[0162] Step S414: the server sends the interactive video to the 3D display device.

[0163] Step S415: the 3D display device displays the interactive video.

[0164] In the embodiment of the present application, by providing the target object with the virtual object in the prefabricated video during the process of displaying the prefabricated video on the 3D display device, an interactive video is generated, and the interaction between the target object and the prefabricated video can be realized without resources from other channels, so that the in-depth and efficient interactive method is expanded in the process of playing the prefabricated video on the 3D display device in a resource-intensive manner, and the playback quality of the prefabricated video is improved. In addition, through the self-behavior operation of the target object, the exploration content (i.e., interactive video) generated for the prefabricated video can be displayed on the 3D display device, thereby improving the viewing interest of the 3D display device and the audience's participation.

[0165] It should be noted that, in other embodiments, the image processing system includes at least a 3D display device and a server. The 3D display device includes a naked-eye 3D screen and may also have an image acquisition device and a processor. The naked-eye 3D screen is used to implement the display steps of the video display in the embodiment of the present application. For the logical steps that require data processing and calculation in the embodiment of the present application, they can be jointly executed by the processor in the 3D display device and the server, that is, they can be executed by the processor in the 3D display device, or the 3D display device can send the data to be processed to the server, and the server can execute the logical steps of data processing and calculation.

[0166] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0167] In recent years, the user base and commercialization of the e-sports market have continued to expand steadily, and the overall industry is accelerating towards industry standardization, diversified cooperation, and e-sports sports. E-sports hotels are an important innovation and attempt by companies to create e-sports life consumption scenarios, and an important exploration for companies to connect and empower the real industry. By creating a new commercial model of "digital IP + urban commercial scenarios", it provides greater imagination space for the development of urban e-sports.

[0168] In the latest commercial exploration, the new model of "virtual human + digital interactive experience + three-dimensional creation of official event IP" was applied for the first time. As a new way of advertising display, the biggest advantage of the 3D large screen (i.e. naked eye 3D screen, a 3D display device) is to use the screen border effect to create naked eye 3D stereoscopic visual effects. Through the combination of technology + aesthetics, culture is output in the form of technology and aesthetics to create a unique visual expression effect. The effect display of naked eye 3D technology has improved the level of advertising and the public's aesthetics to a certain extent. The visual communication effect of naked eye 3D is also difficult to convey by other video ads. In certain specific occasions or when displaying certain specific exhibits, it can firmly grasp the user's vision and bring users a novel and unique visual experience. Adding more interactive gameplay to the visual effect presentation is also a major form of hotel drainage. The more interactive the outdoor advertising is, the easier it is for people to have the idea of ​​sharing. Such outdoor advertising can not only convey information to users, but also use the rich outdoor places where it is located to actively attract audiences to participate. It is the audience who spontaneously spreads in the process of participation and receives advertising information through entertaining experience. Consumers interacting with the 3D large screen can make the interaction more intuitive and provide consumers with better visual effects. It also highlights the unique value of outdoor media. I believe that in the future, this technology and gameplay will be used in more occasions.

[0169] However, for the naked-eye 3D screen in the related art, it is necessary to add interactive gameplay, enhance the atmosphere, improve the interaction between the viewer and the screen, maximize the naked-eye 3D effect, and increase the fun. When the user passes by the naked-eye 3D screen, the display content in the naked-eye 3D screen can change according to the number of people or actions automatically recognized, giving the user an interesting interactive experience.

[0170] Fig.10 3D screen provided by the embodiment of the present application, in the embodiment of the present application, an identification device (i.e., an image acquisition device) is added to the naked-eye 3D screen to enable the user to interact with the screen. At the same time, the user's sense of participation is enhanced. The interactive action can be set according to the displayed content. The interactive action can be a gesture or an action, etc., and even when a certain number of people gather, the change of the screen content can be triggered, giving the user an experience of opening an Easter egg, allowing the user to actively participate, and have a greater sense of immersion and experience.

[0171] Below, the product side of the image processing method provided in the embodiment of the present application will be described by taking the application of the image processing method to a naked-eye 3D screen installed outside an e-sports hotel and playing an advertising video on the naked-eye 3D screen as an example.

[0172] like Fig.11As shown, when no user passes by the naked-eye 3D screen, the screen can automatically play pre-made content, such as a pre-recorded virtual human advertising video (i.e., a pre-made video); when the kinect (a 3D somatosensory camera, i.e., an image acquisition device) in the naked-eye 3D screen detects that people are gathering at the hotel entrance and the distance from the naked-eye 3D screen is within a preset range, the advertising video will be stopped, the current virtual background will be replaced, and the current action of the virtual person in the advertising video will be determined based on the video frame of the current advertising video to generate an animation (i.e., an interactive video) of the virtual person greeting the gathered crowd. For example, the action of the virtual person in the advertising video was originally to face the crowd, and according to the direction of the gathered crowd, the virtual person will be controlled to first turn from facing the crowd to facing the direction of the gathered crowd, and then perform the action of greeting, such as Fig.12 When the user's designated action or the number of people is recognized, the screen content can be triggered to present special effects, such as flower effects when the number of people exceeds a preset threshold, and when a high-five action is recognized, the virtual person responds with a high-five, such as Fig.13 The special effects shown in 131.

[0173] In some embodiments, the virtual person can be controlled to perform different actions according to the number of people entering or leaving the hotel. For example, if the crowd gathers to watch but not check in, the virtual person will show a regretful or disappointed expression, and will show a happy expression when the crowd checks in. After these actions, the advertising video will be played again.

[0174] The hardware devices used in the implementation process of the image processing method provided in the embodiment of the present application include but are not limited to: a naked-eye 3D screen, a camera, a processor, a game engine, etc.

[0175] Fig.14 is an overall flow chart of the image processing method provided in the embodiment of the present application, such as Fig.14 As shown, when no one passes by, the naked eye 3D screen automatically plays the pre-made video (i.e. Fig.14The content of the virtual human prefabricated video). When the audience stands in front of the naked-eye 3D screen, the camera captures the audience's movements or the number of people gathered, captures a two-dimensional image, and inputs the collected two-dimensional image data to the processor for processing. The processor performs image recognition and analysis on the image content captured by the camera. During the recognition and analysis process, the recognized image will be matched with the pictures filed in the picture library to determine whether the recognized image is the specified audience movement or number. If the specified action or number is not reached, it will not affect the screen content. If it is the specified action or number, the judgment result will be transmitted to the game engine. The game engine calculates the picture displayed on the naked-eye 3D screen in real time according to the action or number transmitted by the processor, settles and renders the model of the picture displayed on the naked-eye 3D screen, and transmits the rendered model to the naked-eye 3D screen to present special effects, thereby triggering special effect videos. The audience can control the game characters through their own actions and gestures to achieve interactive gameplay. For example, when the camera captures the audience's designated arm-raising action, the processor will identify and determine the collected graphics, and then transmit the determination results to the game engine. The game engine will solve the model in the naked-eye 3D screen in real time, and render it with the environment, textures, light, etc., to achieve the gameplay effect of the audience interacting with the virtual person in the naked-eye 3D screen, including but not limited to making the virtual person in the naked-eye 3D screen jump and wave, and the body can be tilted left and right to control the direction of travel of the virtual person / screen, etc. In addition, through real-time rendering by the game engine, when the crowd reaches a certain number, the screen content can also be triggered to change.

[0176] The image processing method provided in the embodiment of the present application can realize the exploration content display through the user's own behavior operation, thereby enhancing the fun and audience participation.

[0177] It can be understood that in the embodiments of the present application, the content involving user information, such as user actions, images taken by the user in front of the naked-eye 3D screen, and other information, if it involves data related to user information or corporate information, when the embodiments of the present application are applied to specific products or technologies, it is necessary to obtain user permission or consent, or to blur this information to eliminate the correspondence between this information and the user; and the relevant data collection and processing should be strictly in accordance with the requirements of relevant national laws and regulations when applied in examples, obtain the informed consent or separate consent of the subject of personal information, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the subject of personal information.

[0178] The following is a description of an exemplary structure of the image processing device 354 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2As shown, the image processing device 354 includes: an acquisition model 3541, which is used to acquire an environmental video located within a preset range of the 3D display device through the image acquisition device of the 3D display device during the display of the pre-produced video; an acquisition module 3542, which is used to acquire the current playback screen of the pre-produced video in response to determining that there is a target object within the preset range based on the environmental video and that the current state of the target object meets the preset state index; a determination module 3543, which is used to determine an interactive video that matches the current state of the target object based on the current playback screen, and display the interactive video on the 3D display device.

[0179] In some embodiments, the acquisition module is also used to: perform video recognition on the environmental video to obtain a video recognition result for the preset range; if the video recognition result indicates that the target object is within the preset range, determine the number of the target objects; the number of target objects constitutes the current state of the target object; if the number of target objects is greater than a quantity threshold, determine that the current state of the target object meets the preset state indicator, and obtain the current playback screen of the pre-produced video.

[0180] In some embodiments, the acquisition module is also used to: perform video recognition on the environmental video to obtain a video recognition result for the preset range; if the video recognition result indicates that the target object is within the preset range, determine the action type of the target object; the action type of the target object constitutes the current state of the target object; if the action type of the target object is a preset type, determine that the current state of the target object meets the preset state indicator, and obtain the current playback screen of the pre-produced video.

[0181] In some embodiments, the acquisition module is also used to: perform video frame capture on the environmental video to obtain multiple video frames; perform content consistency analysis on each video frame with images in a preset image library to obtain a consistency analysis result; if the consistency analysis result between the video frame and any image in the preset image library is consistent, obtain attribute information corresponding to the image from the preset image library; determine a video frame recognition result corresponding to the video frame based on the attribute information; and summarize all video frame recognition results in the environmental video to obtain a video recognition result of the environmental video.

[0182] In some embodiments, the acquisition module is also used to: input the environmental video into a pre-trained video object quantity recognition model, identify the number of target objects in the environmental video through the video object quantity recognition model, and obtain the number of target objects for the preset range; input the environmental video into a pre-trained video action recognition model, identify the action of the target objects in the environmental video through the video action recognition model, and obtain the action type of the target objects for the preset range.

[0183] In some embodiments, the determination module is also used to: perform virtual object identification on the current playback screen to obtain virtual objects in the current playback screen; if the number of target objects is greater than a quantity threshold, determine an interactive action that matches the number of target objects for the virtual objects; obtain first rendering data for the virtual object to perform the interactive action; and use the first rendering data to perform image rendering to obtain the interactive video.

[0184] In some embodiments, the environmental video has a preset duration, and a segment of the environmental video is captured every preset duration; the device also includes: a video acquisition module, used to acquire the current environmental video within a current preset duration and a historical environmental video within a historical preset duration connected to the current environmental video and located before the current preset duration; a video recognition module, used to perform video recognition on the current environmental video and the historical environmental video, respectively, and correspondingly obtain the current video recognition result and the historical video recognition result; a quantity extraction module, used to extract the number of target objects from the current video recognition result and the historical video recognition result, respectively, and correspondingly obtain the current number and historical number of target objects; an expression change data determination module, used to determine the expression change data of the virtual object based on the current number and the historical number; a data adding module, used to add the expression change data to the first rendering data.

[0185] In some embodiments, the expression change data determination module is also used to: determine the quantity difference between the current quantity and the historical quantity; if the quantity difference is greater than zero, determine that the virtual object has a first type of expression change process; obtain the expression change data corresponding to the first type of expression change process from a preset expression database; if the quantity difference is less than zero, determine that the virtual object has a second type of expression change process; wherein the first type of expression change process and the first type of expression change process are used to represent expression change processes corresponding to two different emotions; obtain the expression change data corresponding to the second type of expression change process from the preset expression database.

[0186] In some embodiments, the device also includes: a position identification module, which is used to identify the position of the target object based on the environmental video to obtain the position change information of the target object; an analysis module, which is used to determine that the virtual object has a third type of expression change process if the position change information of the target object meets a preset position change condition; the expression change data determination module is also used to obtain expression change data corresponding to the third type of expression change process from a preset expression database; the data adding module is also used to add the expression change data corresponding to the third type of expression change process to the first rendering data.

[0187] In some embodiments, the determination module is also used to: perform virtual object identification on the current playback screen to obtain the virtual object in the current playback screen; if the action type of the target object is a preset type, determine an interactive action matching the action type from a preset action library; obtain second rendering data of the virtual object performing the interactive action; and use the second rendering data to perform image rendering to obtain the interactive video.

[0188] In some embodiments, the device further includes: a detection module for detecting whether a new interactive video is generated after playing the interactive video; and a video playback module for continuing to play the pre-produced video if no new interactive video exists.

[0189] It should be noted that the description of the device of the embodiment of the present application is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment, so it is not repeated. For technical details not disclosed in the embodiment of the device, please refer to the description of the method embodiment of the present application for understanding.

[0190] The embodiment of the present application provides a computer program product, which includes an executable instruction, which is a computer instruction; the executable instruction is stored in a computer-readable storage medium. When a processor of an electronic device reads the executable instruction from the computer-readable storage medium and the processor executes the executable instruction, the electronic device executes the method described in the embodiment of the present application.

[0191] The embodiment of the present application provides a storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the method provided by the embodiment of the present application, for example, Figure 3 The method shown.

[0192] In some embodiments, the storage medium can be a computer-readable storage medium, for example, a ferroelectric random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPR OM), a flash memory, a magnetic surface memory, an optical disk, or a compact disk read-only memory (CD-ROM), etc.; it can also be various devices including one or any combination of the above memories.

[0193] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0194] As an example, executable instructions may, but need not correspond to, a file in a file system, may be stored as part of a file storing other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions). As an example, executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed at multiple locations and interconnected by a communication network.

[0195] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: In the process of displaying the pre-made video on the display device, an environment video within a preset range of the display device is captured by an image capture device of the display device; In response to determining that there is a target object within the preset range based on the environmental video, and that a current state of the target object satisfies a preset state indicator, obtaining a current playback screen of the pre-produced video; Based on the current playing screen, an interactive video matching the current state of the target object is determined, and the interactive video is displayed on the display device.

2. The method according to claim 1, characterized in that In response to determining that there is a target object within the preset range based on the environmental video and that the current state of the target object meets a preset state indicator, obtaining the current playback screen of the pre-produced video includes: Performing video recognition on the environmental video to obtain a video recognition result for the preset range; If the video recognition result indicates that the target object exists within the preset range, determining the number of the target objects; the number of the target objects constitutes the current state of the target objects; If the number of the target objects is greater than the number threshold, it is determined that the current state of the target objects meets the preset state indicator, and the current playback screen of the pre-produced video is obtained.

3. The method according to claim 1, characterized in that In response to determining that there is a target object within the preset range based on the environmental video and that the current state of the target object meets a preset state indicator, obtaining the current playback screen of the pre-produced video includes: Performing video recognition on the environmental video to obtain a video recognition result for the preset range; If the video recognition result indicates that the target object exists within the preset range, determining the action type of the target object; the action type of the target object constitutes the current state of the target object; If the action type of the target object is a preset type, it is determined that the current state of the target object meets the preset state indicator, and the current playback screen of the pre-produced video is obtained.

4. The method according to claim 2 or 3, characterized in that: The performing video recognition on the environment video to obtain a video recognition result for the preset range includes: Performing video frame interception on the environment video to obtain multiple video frames; Perform content consistency analysis on each video frame and the images in the preset image library to obtain consistency analysis results; If the consistency analysis result between the video frame and any image in the preset image library is consistent, acquiring attribute information corresponding to the image from the preset image library; Determine a video frame recognition result corresponding to the video frame based on the attribute information; The recognition results of all video frames in the environmental video are summarized to obtain a video recognition result of the environmental video.

5. The method according to claim 2 or 3, characterized in that: The performing video recognition on the environment video to obtain a video recognition result for the preset range includes: Inputting the environment video into a pre-trained video object quantity recognition model, identifying the number of target objects in the environment video through the video object quantity recognition model, and obtaining the number of target objects for the preset range; The environment video is input into a pre-trained video action recognition model, and the action of the target object in the environment video is recognized by the video action recognition model to obtain the action type of the target object in the preset range.

6. The method according to claim 2, characterized in that The step of determining, based on the current playing screen, an interactive video that matches the current state of the target object comprises: Performing virtual object recognition on the current playing picture to obtain the virtual object in the current playing picture; If the number of the target objects is greater than a number threshold, determining an interactive action matching the number of the target objects for the virtual object; Acquire first rendering data of the virtual object performing the interactive action; The first rendering data is used to perform image rendering to obtain the interactive video.

7. The method according to claim 6, characterized in that The environment video has a preset duration, and a segment of the environment video is captured every preset duration; the method further includes: Acquire a current environment video within a current preset duration and a historical environment video within a historical preset duration that is connected to the current environment video and is located before the current preset duration; Performing video recognition on the current environment video and the historical environment video respectively, and obtaining a current video recognition result and a historical video recognition result correspondingly; Extracting the number of target objects from the current video recognition result and the historical video recognition result respectively, and obtaining the current number and the historical number of target objects accordingly; Determining expression change data of the virtual object based on the current number and the historical number; The expression change data is added to the first rendering data.

8. The method according to claim 7, characterized in that The step of determining expression change data of the virtual object based on the current number and the historical number includes: determining a quantity difference between the current quantity and the historical quantity; If the quantity difference is greater than zero, it is determined that the virtual object has a first type of expression change process; Acquire expression change data corresponding to the first type of expression change process from a preset expression database; If the quantity difference is less than zero, it is determined that the virtual object has a second type of expression change process; wherein the first type of expression change process and the first type of expression change process are used to represent expression change processes corresponding to two different emotions; Acquire expression change data corresponding to the second type of expression change process from the preset expression database.

9. The method according to claim 6, characterized in that The method further comprises: Based on the environmental video, position recognition is performed on the target object to obtain position change information of the target object; If the position change information of the target object meets a preset position change condition, determining that the virtual object has a third type of expression change process; Acquire expression change data corresponding to the third type of expression change process from a preset expression database; Adding expression change data corresponding to the third type of expression change process to the first rendering data.

10. The method according to claim 3, characterized in that: The step of determining, based on the current playing screen, an interactive video that matches the current state of the target object includes: Performing virtual object recognition on the current playing picture to obtain the virtual object in the current playing picture; If the action type of the target object is a preset type, determining an interactive action matching the action type from a preset action library; Acquire second rendering data of the virtual object performing the interactive action; The second rendering data is used to perform image rendering to obtain the interactive video.

11. The method according to claim 1, characterized in that The method further comprises: After playing the interactive video, detecting whether a new interactive video is generated; If there is no new interactive video, continue to play the pre-made video.

12. An image processing device, characterized in that: The device comprises: An acquisition model, used for acquiring an environment video within a preset range of the display device through an image acquisition device of the display device during display of a pre-made video; An acquisition module, configured to acquire a current playback screen of the pre-produced video in response to determining that a target object exists within the preset range based on the environmental video and that a current state of the target object satisfies a preset state indicator; A determination module is used to determine an interactive video that matches the current state of the target object based on the current playback screen, and display the interactive video on the display device.

13. An electronic device, characterized in that: include: A memory for storing executable instructions; A processor, configured to implement the image processing method according to any one of claims 1 to 11 when executing the executable instructions stored in the memory.

14. A computer-readable storage medium, characterized in that: Executable instructions are stored, which are used to cause a processor to execute the executable instructions to implement the image processing method according to any one of claims 1 to 11.

Images